VIPE: editing the image beats editing the prompt for video reasoning models
Researchers led by Robert Geirhos studied whether video models, which are increasingly used as foundation models for visual reasoning, benefit from a visual counterpart to text-based prompt engineering. Their method, called VIPE (visual prompt engineering), automatically modifies the task image itself rather than the text instructions given to the model. The paper's example task is a physics reasoning problem: given a scene showing a ball approaching a set of obstacles, the model must predict where the ball lands. Instead of rewording the question, VIPE edits the input image, for instance turning an abstract, sketch-like version of the scene into a photorealistic one using an image editing model, before handing it to the video model. Across the reasoning tasks tested, this image-level intervention improved performance. The authors report that VIPE was more effective at boosting video reasoning than two established alternatives: classic text-based prompt engineering and test-time scaling (giving the model more compute or samples at inference). Their conclusion is that video models respond more strongly to changes in what they see than to changes in how the task is phrased, making visual prompt engineering a comparatively simple and compute-efficient way to raise visual reasoning performance. The paper's project page hosts example videos of the technique.
Key facts
- VIPE (visual prompt engineering) improves video model reasoning by editing the input image rather than the text prompt.
- Example task: predicting where a ball lands after passing obstacles, using an image editing model to convert a sketch-like scene into a photorealistic one before feeding it to the video model.
- The authors report VIPE outperforms both classic text-based prompt engineering and test-time scaling on the tasks tested.
- Example videos are hosted on the project page at visual-prompt-engineering.github.io.
Why it matters
Prompt engineering for language models is well established: how you phrase a question changes what a model outputs. This paper asks the same question for video models acting as visual reasoners, and finds the more effective lever is not the wording of the task but the image the model is shown. That reframes what counts as a 'prompt' for visual foundation models.
Who it affects
The result is aimed at researchers and engineers building or evaluating video and visual reasoning models, particularly anyone currently relying on text-prompt tweaks or extra inference-time compute to squeeze out better performance on visual tasks.
How to use it
The described workflow is to take the task image and run it through an image editing model to change its visual style, for example converting an abstract sketch into a photorealistic rendering, before passing it to the video model for reasoning. The paper and example videos are available on the project's public page; the text does not mention pricing, a released tool, or licensing terms.
How solid is it
The claims come from the authors' own paper describing experiments across multiple video reasoning tasks, with the physics example (predicting a ball's landing spot after obstacles) used as the illustrative case. The source text does not give exact benchmark scores or task counts, so the size and generality of the improvement cannot be quantified beyond the authors' summary.
Risks and caveats
The available text is a project summary rather than the full paper, so specifics on datasets, baselines, and exact performance margins are not given here. The author list also likely extends beyond Robert Geirhos, who is named as the point of contact.
“an abstract sketch-like scene can be turned into a photorealistic version with a simple call to an image editing model”
— project paper, described by Robert Geirhos and co-authors