π^4 model recovers 4D scene geometry and motion with about 0.5B parameters

π^4 model recovers 4D scene geometry and motion with about 0.5B parameters

A new paper presents π^4, which the authors call a unified foundation model for 4D scene perception. It takes uncalibrated RGB observations "in the wild": a single image, an unordered collection of images, or a video sequence. From these it recovers dense geometry, camera poses and metric scale. For multi-image inputs it also produces 2D/3D point tracks and moving-object masks.

The architecture couples two parts. A geometry branch reconstructs the scene. A geometry-conditioned motion branch predicts image-space correspondences, and these are lifted into 3D using the reconstructed geometry of the target view. The full geometry-and-motion system shares a backbone and contains approximately 0.5B parameters, which the authors say is less than half the size of most baselines.

On results, the abstract says that across diverse benchmarks π^4 achieves the best overall average rank in geometry evaluation. For single- and multiview 2D/3D point tracking and moving-object segmentation it reports competitive performance, which is a weaker claim than best. The abstract names no individual benchmarks, baselines or scores.

The model produces joint reconstructions and tracks in one forward pass. That includes large-scale scenes with thousands of input images, even though training used contexts of at most 64 views. The paper also presents the data and training recipe behind these capabilities, along with empirical studies of key design choices. Code and models are released at https://github.com/yyfz/Pi4.

Key facts

  • π^4 recovers dense geometry, camera poses and metric scale from a single image, an unordered image collection or a video, with no calibration needed.
  • For multi-image inputs it also outputs 2D/3D point tracks and moving-object masks, using a geometry branch plus a geometry-conditioned motion branch.
  • The authors report the best overall average rank in geometry evaluation, and competitive (not best) results in point tracking and moving-object segmentation.
  • The full system has approximately 0.5B parameters, less than half the size of most baselines.
  • It handles scenes with thousands of input images in one forward pass despite training on contexts of at most 64 views; code and models are on GitHub.

Why it matters

The paper targets both reconstructing a scene and tracking what moves in it. π^4 puts geometry, camera pose, metric scale, point tracks and moving-object masks into one model that gives joint output in a single forward pass. It does this at roughly 0.5B parameters, less than half the size of most baselines, which makes a one-model approach look practical. The claim that it scales to thousands of input images after training on at most 64 views is also notable.

Who it affects

The work is aimed at researchers and engineers in 3D reconstruction, camera pose estimation, point tracking and motion segmentation. It takes in-the-wild, uncalibrated RGB input, so it speaks to anyone working with ordinary photos or video rather than calibrated capture rigs.

How to use it

Code and models are available at https://github.com/yyfz/Pi4. The model accepts a single image, an unordered collection of images, or a video sequence. Tracks and moving-object masks come out for multi-image inputs. The paper also describes the data and training recipe and includes studies of key design choices, which is useful for anyone trying to reproduce or adapt the approach. No license for the code and models is stated.

How solid is it

The evidence here is the paper's own abstract: the authors claim the best overall average rank in geometry evaluation across diverse benchmarks. The best-rank claim applies to geometry only. For point tracking and moving-object segmentation the wording is "competitive". No benchmark names, baseline names or numeric scores are given, so the strength of the lead cannot be judged from this text. The released code and models allow independent checking.

Risks and caveats

The headline result is an average rank, which does not say how large the margins are or how many baselines were compared. The 0.5B figure compares parameter count only, not accuracy. The abstract gives no figures for training data, compute, latency or inference speed, and no exact number for the "thousands" of images handled at inference. It also does not state a license for the code and models.

“It couples a geometry branch with a geometry-conditioned motion branch, which predicts image-space correspondences that are lifted into 3D using reconstructed target-view geometry.”

— π^4 paper abstract