Paper probes whether visual foundation models keep a persistent spatial state
An observer sees only part of the world around it, yet has to keep track of what lies outside the current view. The paper argues that spatial perception is therefore more than recognising individual images: it calls for a persistent spatial state that is updated under self-motion and used to guide action. The question it asks is whether visual foundation models provide representations that support such a state.
To test this, the authors use frozen-feature probes on ten representations, under streaming egocentric observations of static indoor environments. They organise the test around four levels of spatial capability: spatial structure, object persistence, predictive spatial update, and actionability.
The resulting capability profiles vary across tasks, network layers and prefix lengths. A strong readout of the visible scene does not ensure accurate object persistence or future-feature retrieval. Matched controls separate readout accuracy from reliance on visual history and motion.
On prediction, native future prediction retains recoverable spatial structure, but a spatial reader trained on real features does not transfer to predicted features unchanged. On action, goal-dependent candidate selection does not establish reliable continuous action utility: predicted actions respond to goal changes, yet supplying the correct goal does not reliably improve progress.
Across the four levels, object persistence and actionability build on camera-pose readout, whereas feature prediction builds on visible-scene readout. The authors conclude that spatial representations should be evaluated not only by what can be decoded from them, but by whether that information is maintained beyond observation, updated predictively, and used for goal-directed action.
Key facts
- The study probes ten representations with frozen-feature probes on streaming egocentric observations of static indoor environments.
- It covers four levels of spatial capability: spatial structure, object persistence, predictive spatial update and actionability.
- Strong readout of the visible scene does not ensure accurate object persistence or future-feature retrieval.
- A spatial reader trained on real features does not transfer unchanged to predicted features.
- Supplying the correct goal does not reliably improve progress, even though predicted actions respond to goal changes.
Why it matters
Many benchmarks judge a visual model by what can be decoded from an image or a clip. This paper points to a different test: whether the model keeps a spatial state that persists beyond the current view, is updated as the observer moves, and can guide action. Its finding is that good decoding of what is visible does not carry over to remembering unseen objects or retrieving future features.
Who it affects
The work is aimed at researchers who build or evaluate visual foundation models and spatial representations, especially for settings where an observer moves through an environment and must act in it. The abstract frames the question around egocentric, streaming observation.
How to use it
The practical takeaway is a way of evaluating. The authors motivate testing spatial representations on whether information is maintained beyond observation, updated predictively and used for goal-directed action, rather than on decodability alone. They also report that results differ by task, network layer and prefix length, so a single probe at a single layer would give an incomplete picture.
How solid is it
This is a preprint listed on Hugging Face Papers, and the account here rests on its abstract. The study uses matched controls to separate readout accuracy from reliance on visual history and motion. The abstract gives no numerical results.
Risks and caveats
The setting is streaming egocentric observation of static indoor environments, and the claims should be read within that scope. The ten representations are not named in the abstract, so which models behave which way cannot be told from it. The findings are diagnostic: they show where representations fall short, not how to fix it.