Glonce

Self-Geometry fixes multi-view errors in vision foundation models

researchThe Glonce editorsHugging Face🔥66Glonce rating66Impact60Novelty65Relevance90Surprise45Players25Credibility80
Self-Geometry fixes multi-view errors in vision foundation models

Vision Foundation Models (VFMs) predict depth, camera pose and pointmap in a single forward pass, without per-scene optimization, and generalize well as a result. But enforcing explicit multi-view geometric consistency, for example through bundle adjustment, is computationally costly, so it is not applied during VFM pretraining, and geometric inconsistency across different views of the same scene can arise.

Prior work tried to fix this at test time using implicit self-consistency signals derived from the model's own outputs, such as pointmaps or internal features. The authors say this delivers only limited performance gains, especially on scenes where the pretrained VFM is highly inaccurate to begin with.

The researchers propose Self-Geometry, a plug-and-play test-time adaptation (TTA) pipeline that instead imposes explicit multi-view geometric constraints directly, using 2D pixel correspondences between views as pseudo ground-truth. It combines three parts: Geometric Disentanglement Optimization, which combines Multi-View Consistency and Epipolar Consistency losses with a technique called Gradient Disentanglement to stop the two losses' gradients from conflicting; Frame Angular-Neighbor, a view sampler based on SO(3) geodesic distances that decides which frame pairs the constraints are lightly imposed on; and Lightweight TTA, which adapts the underlying VFM through LoRA rather than full fine-tuning.

The authors report that Self-Geometry achieves consistent improvements in both pose and geometry estimation across six VFMs, VGGT, π^3, and four sizes of DA3 (Giant, Large, Base, Small), tested on four benchmarks: 7Scenes, ETH3D, ScanNet++ and HiRoom.

Key facts

  • Self-Geometry is a plug-and-play test-time adaptation pipeline that enforces explicit multi-view geometric consistency in Vision Foundation Models, using 2D pixel correspondences as pseudo ground-truth rather than requiring real ground-truth geometry.
  • It combines three components: Geometric Disentanglement Optimization (Multi-View Consistency and Epipolar Consistency losses combined via Gradient Disentanglement), Frame Angular-Neighbor (an SO(3) geodesic-distance view sampler) and Lightweight TTA (LoRA-based adaptation).
  • Prior test-time methods relied on implicit self-consistency signals from a model's own outputs, which the authors say give limited gains, especially where the pretrained VFM is highly inaccurate.
  • The method was tested across six VFMs, VGGT, π^3, and the Giant, Large, Base and Small sizes of DA3, on four benchmarks: 7Scenes, ETH3D, ScanNet++ and HiRoom.
  • The authors report consistent improvements in both pose and geometry estimation across all six models and four benchmarks, though the abstract gives no numeric size for the gains.

Why it matters

VFMs such as VGGT and DA3 predict depth, camera pose and pointmaps in a single forward pass, without per-scene optimization, and generalize well as a result. But because enforcing full multi-view geometric consistency, for example through bundle adjustment, is too costly to apply during pretraining, their outputs can be geometrically inconsistent across different views of the same scene. That limits how reliable these models are for tasks where several views of a scene need to agree with each other. Self-Geometry addresses this by enforcing consistency explicitly at test time rather than during pretraining, without requiring per-scene ground truth.

Who it affects

Researchers and practitioners building or evaluating Vision Foundation Models for camera pose and 3D geometry estimation, specifically anyone working with models such as VGGT, π^3 or DA3. Because Self-Geometry is plug-and-play, it is meant to work on top of an already-pretrained VFM rather than requiring it to be retrained from scratch.

How to use it

Self-Geometry runs at test time on top of an already-pretrained VFM. It uses 2D pixel correspondences between views as pseudo ground-truth instead of real ground-truth geometry, then adapts the model through three combined pieces: Geometric Disentanglement Optimization (Multi-View Consistency and Epipolar Consistency losses, kept from conflicting through Gradient Disentanglement), Frame Angular-Neighbor (a view sampler based on SO(3) geodesic distance that picks which frame pairs the constraints are imposed on) and Lightweight TTA (adapting the VFM via LoRA rather than full fine-tuning). The abstract does not mention a release date, code availability or dataset release.

How solid is it

The authors evaluated Self-Geometry across six VFMs, VGGT, π^3 and the Giant, Large, Base and Small sizes of DA3, on four benchmarks: 7Scenes, ETH3D, ScanNet++ and HiRoom. They report consistent improvements in both pose and geometry estimation across all of them. The abstract does not give the numeric size of the improvement, and it does not name the specific prior methods Self-Geometry is compared against, beyond calling them "prior work." It also does not name the paper's authors or institutions.

Risks and caveats

Because the abstract omits both the magnitude of the reported gains and the identity of the baselines, it is not possible to judge from this text alone how large an improvement Self-Geometry provides or how it compares with specific competing methods. No code or dataset release is mentioned, so reproducing the results is not currently possible from the information given here.

“enforcing explicit multi-view geometric consistency, e.g., through bundle adjustment, is computationally costly and is thus not imposed during VFM pretraining, so such inconsistency can arise”

— the paper's abstract