Video-Coder trains a coding agent to draft 3D scenes for video generators
Video generation has improved a lot in visual quality and temporal consistency, but the paper's authors say that producing videos with precise spatial structure, coherent motion and controllable scene composition remains challenging. Their answer is Video-Coder, which they call the first attempt to train a code-driven video generation agent.
The agent works by autonomously writing and executing code that builds a dynamic 3D scene, called a Draft Video. That draft supplies structure and motion signals to downstream video generators, so the generator does not have to invent layout and movement from scratch.
To make the approach trainable, the authors built a dedicated data pipeline and curated two datasets, Video-Coder-SFT-6k and Video-Coder-RL-2k. Both contain diverse prompts and ground-truth reference videos. They also developed the Video-Coder Harness, a unified interactive environment that combines code execution, 3D scene rendering and execution feedback, used for both agent training and inference.
Training has two stages. First comes supervised fine-tuning (SFT). Then comes harness-native agentic reinforcement learning with dual reward feedback, combining execution-based rewards and video-based rewards. The authors add an annealing strategy that gradually reduces the execution reward over time, so that in later training stages the model relies more on the video reward.
On results, the paper says experiments show significant gains: Video-Coder improves Minimax-H3 by approximately 7 points on PhyGenBench and 24 points on VBench-2.0. The authors say they fully open-source their data, models and code, and hope the work can serve as an open foundation for coding agents in video generation.
Key facts
- Video-Coder is presented as the first attempt to train a code-driven video generation agent that writes and executes code to build a dynamic 3D scene, the Draft Video, giving structure and motion signals to downstream video generators.
- Two curated datasets, Video-Coder-SFT-6k and Video-Coder-RL-2k, plus the Video-Coder Harness (code execution, 3D rendering, execution feedback) support training and inference.
- Training is SFT followed by agentic reinforcement learning with execution-based and video-based rewards; an annealing strategy gradually reduces the execution reward so the video reward dominates later.
- Reported gains for Minimax-H3: approximately 7 points on PhyGenBench and 24 points on VBench-2.0.
- The authors say they fully open-source their data, models and code.
Why it matters
The paper targets a known weak spot in video generation: precise spatial structure, coherent motion and controllable scene composition. Instead of asking a generator to get all of this right on its own, Video-Coder has an agent write code that builds a 3D scene first, and the generator follows that draft. The authors present this as the first attempt to train such a code-driven agent, and frame the release as a possible open foundation for coding agents in video generation.
Who it affects
Mainly researchers and engineers working on video generation, especially those who care about physical plausibility, motion and scene control. Anyone building agent training setups with code execution and rendering feedback may also find the harness and reward design relevant.
How to use it
The authors say they fully open-source the data, models and code, including the Video-Coder-SFT-6k and Video-Coder-RL-2k datasets and the Video-Coder Harness. No links or licences for the released data, models and code are given in the text, so check the paper page for where they live and under what terms. In practice the model's Draft Video is passed to a downstream video generator as structure and motion guidance.
How solid is it
The claims come from the paper's own abstract. The authors report approximately 7 points on PhyGenBench and 24 points on VBench-2.0 for Minimax-H3, and describe the gains as significant. No baseline scores on either benchmark are given, so the resulting absolute scores are unknown. No comparison against other video generation systems is given. The source does not say what Minimax-H3 is beyond the name, nor which downstream video generators were used besides it.
Risks and caveats
The 7 and 24 point figures are stated only as points; the source does not say whether they are absolute or relative gains, or whether they are a mean or a best case. No compute cost, training time, model size or base model for Video-Coder is stated. The open-source release is a statement of intent in the abstract, and the authors' hope that this can serve as an open foundation is not a demonstrated result.
“We hope this work can serve as an open foundation for coding agents in video generation”
— Video-Coder paper authors