Glonce

4DAnyone reconstructs 4D humans from a single casual video

researchThe Glonce editorsHugging Face🔥56Glonce rating56Impact65Novelty72Relevance98Surprise45Players38Credibility83
4DAnyone reconstructs 4D humans from a single casual video

Yudong Jin and co-authors present 4DAnyone, a framework for reconstructing 4D humans, a moving three-dimensional representation of a person viewable from any angle at any point in time, from a single uncalibrated, casual video shot with one camera. The method works in two stages: it first generates reconstruction-grade, multiview-consistent videos of the subject from many new viewpoints, then lifts that set of videos into a 4D Gaussian Splatting (4DGS) representation.

The paper's starting point is a specific failure in existing tools. Camera-controlled video diffusion models can already synthesize a plausible novel-view video, footage from an angle the source camera never captured, but the authors find that this consistency breaks down once generation is scaled to the tens of target views a full 4DGS reconstruction actually needs.

They trace the failure to what they call a bounded-attention-context problem. A single diffusion transformer (DiT) forward pass can only handle so many views at once, so once the target view count exceeds that limit, the views have to be split into separate groups, and that split creates two coupled bottlenecks. On the reference side, the model conditions on every previously generated view, and that conditioning grows as O(N): as more views accumulate, cross-view appearance guidance gets weaker. On the target side, once views are split into disjoint groups, those groups cannot exchange information with each other directly, so the overall structure drifts apart across groups.

4DAnyone addresses both bottlenecks with two techniques designed to work together. Reference Context Packing (RCP) compresses the growing set of reference views into a fixed-length, mixed-resolution context, cutting reference-context complexity from O(N) down to O(1): the conditioning cost stops growing no matter how many views have already been generated. Target Context Routing (TCR) rotates which target views are grouped together as the model denoises the output, sharing context across groups during the early, high-noise steps and stabilizing fine detail during the later, low-noise steps.

To train the system, the authors built a new dataset, MVGameHuman, generated with an in-house game engine, and combined it with existing light-stage capture and in-the-wild video. Tested on DNA-Rendering and DyMVHumans, 4DAnyone is reported to outperform prior methods in both novel-view video quality and downstream 4DGS reconstruction, with what the authors describe as robust generalization to in-the-wild footage. The comparison is stated qualitatively in the abstract, without published accuracy or error numbers, and without naming the prior methods it was measured against.

Key facts

  • 4DAnyone reconstructs 4D humans, a moving 3D representation of a person viewable from any angle over time, from a single uncalibrated, casual video, by generating consistent novel-view videos and lifting them into a 4D Gaussian Splatting (4DGS) representation.
  • The authors diagnose a bounded-attention-context problem: once the number of target views exceeds what one diffusion-transformer pass can handle, the views split into groups, and conditioning on every prior view grows as O(N), weakening cross-view appearance guidance.
  • Reference Context Packing (RCP) compresses the growing set of reference views into a fixed-length, mixed-resolution context, cutting reference-context complexity from O(N) to O(1).
  • Target Context Routing (TCR) rotates which target views are grouped together during denoising, sharing context across groups at high-noise steps and stabilizing detail at low-noise steps, to stop the global structural drift that disjoint groups otherwise cause.
  • The authors built a new dataset, MVGameHuman, from an in-house game engine, combined it with light-stage and in-the-wild footage for training, and report that 4DAnyone outperforms prior methods on the DNA-Rendering and DyMVHumans benchmarks, with robust in-the-wild generalization.

Why it matters

Turning an ordinary, single-camera video into a full 4D reconstruction of a person, viewable from any angle at any time, has increasingly relied on video diffusion models to synthesize the extra camera angles that were never actually filmed. The problem this paper addresses is that those models look right for one new view but break down once a full 4DGS reconstruction needs tens of consistent views at once, because a single diffusion transformer pass cannot process that many views together and has to split them into groups. 4DAnyone's contribution is naming that failure precisely as a bounded-attention-context problem with two coupled bottlenecks, and offering two mechanisms, Reference Context Packing and Target Context Routing, that keep the growing set of reference views cheap to condition on and keep separate groups of target views from drifting apart. It is a scaling fix for turning casual, single-camera footage into a usable 4D reconstruction, not a new capability introduced from nothing.

Who it affects

The direct audience is researchers and engineers building video-diffusion-based pipelines for novel-view synthesis and 4D human reconstruction, especially anyone who has hit the same wall the authors describe: a camera-controlled video diffusion model producing a plausible single new-view video but losing consistency once tens of views are needed for a full 4DGS reconstruction. The authors present 4DAnyone as a research framework and point to a project page, https://4danyone.github.io, for video results and source code.

How to use it

4DAnyone is presented as a research framework. The authors point readers to a project page, https://4danyone.github.io, for video results and source code: the entry point for anyone wanting to inspect the method, reproduce the experiments on DNA-Rendering and DyMVHumans, or build on it for their own 4D human reconstruction from casual video.

How solid is it

Beyond the architecture, the authors also address the training-data problem: they built a new dataset, MVGameHuman, using an in-house game engine, and combined it with existing light-stage capture and in-the-wild video for training. On the DNA-Rendering and DyMVHumans benchmarks, 4DAnyone is reported to outperform prior methods in both novel-view video quality and downstream 4DGS reconstruction, with what the authors describe as robust generalization to in-the-wild footage.

Risks and caveats

The claim that 4DAnyone outperforms prior methods is stated only qualitatively in the abstract: it gives no specific accuracy, quality, or reconstruction-error numbers, and it does not name the prior methods used for comparison. It also gives no exact count of target views, only the phrase tens of target views, and it discloses no training compute, model size, dataset size, or hardware, so the resource cost behind the results is not stated. The text names no institutional affiliations; it speaks only in the first person plural throughout.