4DAnyone rebuilds moving humans in 4D from one video
4DAnyone is a framework for reconstructing a full 4D representation of a human, meaning a 3D shape that also captures how the person moves over time, starting from an uncalibrated monocular video: ordinary footage shot from a single camera with no known position or calibration data. The system works by first generating a set of new, synthetic video views of the same subject from many virtual camera angles, made to stay consistent with one another, and then feeding those views into 4D Gaussian Splatting (4DGS), a technique that assembles many-view video into a reconstructed, moving 3D scene.
The authors identify a specific failure in prior camera-controlled video diffusion models used for this task: these models can synthesize a plausible video from any single new viewpoint, but when the number of viewpoints scales up to the tens that 4DGS reconstruction requires, the generated views stop matching each other. The authors trace this to what they call a bounded-attention-context problem. A diffusion transformer (DiT) can only process a limited number of views in one forward pass, so once the target number of viewpoints exceeds that limit, the views must be generated in separate groups. That split creates two coupled bottlenecks: conditioning each new group on every view generated so far means the reference context grows with the number of views already produced, weakening how well appearance carries across views, while the separate groups cannot exchange information with each other, letting the overall structure of the reconstruction drift as more groups are added.
4DAnyone addresses both bottlenecks with two paired designs. Reference Context Packing (RCP) compresses the growing set of reference views into a fixed-length context built from mixed resolutions, holding the cost of conditioning on prior views constant instead of letting it grow with the number of views. Target Context Routing (TCR) rotates which target views are grouped together as the video diffusion process denoises its output: at high-noise steps early in denoising, groups are mixed so information can flow between them; at low-noise steps later on, groupings stabilize to lock in fine detail.
To train the system, the authors built a new dataset called MVGameHuman using their own in-house game engine, and combined it with existing light-stage capture data and in-the-wild video. In experiments on the DNA-Rendering and DyMVHumans benchmarks, the authors report that 4DAnyone outperforms prior methods on both the quality of the generated novel-view videos and on the downstream 4DGS reconstruction built from them, with robust generalization to in-the-wild footage beyond the benchmark data. The paper does not give the specific numeric scores behind that comparison, name the authors, or state a publication date. A project page with video results and source code is available at https://4danyone.github.io.
Key facts
- 4DAnyone reconstructs a moving 4D human, a 3D shape plus its motion over time, from a single uncalibrated monocular video by generating consistent novel-view videos and lifting them into 4D Gaussian Splatting.
- The authors diagnose a bounded-attention-context problem: splitting the tens of target views a 4DGS reconstruction needs across multiple diffusion-transformer passes breaks consistency, since growing reference context weakens cross-view appearance guidance and separated groups cannot exchange information, causing structural drift.
- Reference Context Packing (RCP) holds reference-view context at a fixed size and constant cost; Target Context Routing (TCR) rotates which target views are grouped together during denoising, sharing information across groups early and stabilizing detail late.
- The authors built a new synthetic training dataset, MVGameHuman, from their in-house game engine, and combined it with light-stage and in-the-wild video.
- On the DNA-Rendering and DyMVHumans benchmarks, the authors report 4DAnyone beats prior methods on novel-view video quality and downstream 4DGS reconstruction, with robust in-the-wild generalization; the paper gives no numeric scores.
Why it matters
Turning ordinary, casually shot video into a full 4D reconstruction of a moving person, rather than requiring a calibrated multi-camera rig or a light stage, would make dynamic human digitization far more accessible. The paper's core contribution is not the idea of using video diffusion for this, which prior work already attempted, but a specific fix for why those earlier systems broke down once the number of viewpoints needed for 4D Gaussian Splatting reconstruction scaled into the tens: the bounded-attention-context problem the authors describe, where splitting generation into groups either lets appearance guidance decay or lets structure drift between groups. Addressing that failure mode directly is what the paper presents as its advance over prior camera-controlled video diffusion approaches.
Who it affects
The work is aimed at researchers building novel-view synthesis and 4D human reconstruction systems, and by extension at fields that depend on turning video into usable 3D or 4D assets: game and VFX pipelines, digital-human and avatar creation, and AR or VR content production. It also touches researchers building synthetic training data, since the authors' own MVGameHuman dataset was produced with an in-house game engine rather than captured live.
How to use it
The authors have published a project page with video results and source code at https://4danyone.github.io. The source text gives no pricing, license terms, or usage restrictions, so none should be assumed.
How solid is it
The claims of outperforming prior methods rest on experiments the authors ran themselves on two named benchmarks, DNA-Rendering and DyMVHumans, covering both novel-view video quality and downstream 4DGS reconstruction, plus a claim of robust generalization to in-the-wild video. The source text does not include the specific numeric scores behind those comparisons, so the size of the improvement cannot be assessed from what is given here; readers who need the numbers should consult the paper or code directly.
Risks and caveats
The retelling here is limited by what the source text states: it gives no author names or institutional affiliation, no publication or submission date, no specific count of target views beyond a general reference to "tens" of them, and no quantitative performance metrics for the benchmark comparisons. As with any single paper's self-reported results, the outperformance claims have not been independently verified in the text available here.
“Existing camera-controlled video diffusion models synthesize plausible novel-view videos but fail to maintain consistency when scaled to the tens of target views required for 4DGS reconstruction.”
— 4DAnyone paper