UrbanGround finds MLLM agents fail to sustain city-scale navigation
A team of researchers has proposed UrbanGround, which they describe as the first sandbox that makes a specific question testable: does an MLLM agent's local reading of a street scene stay useful once the agent actually starts moving through a city? The sandbox is a physically constrained, real-scale replica of Hong Kong, assembled from the city's territory-wide 3D geospatial data rather than a simplified or synthetic layout.
UrbanGround supports closed-loop interaction from a first-person view and includes an interactive map for navigation. Agents enter the 3D city directly and explore it from street level, the way a pedestrian would, rather than working from a single fixed image.
The authors structure their analysis around three research questions, each building on the last. First, can an agent ground a local scene well enough to answer spatial questions after actively observing it? Second, does that grounding still support navigation once destinations sit farther away and are described less explicitly? Third, does the resulting behavior hold up when routes change or pedestrians move through the scene?
The results split along that progression. Contemporary MLLM agents usually handle the atomic pieces well, with useful visual recognition and short-range spatial reasoning. Orientation and pedestrian-aware movement do not fare as well; both remain unreliable. The paper describes the agents' central failure as showing up over extended exploration, where these local abilities fail to compose into sustained, goal-directed behavior and small errors accumulate without the agent correcting them.
The authors frame UrbanGround as a resource for the field rather than a one-off benchmark: they say they hope it will support broader study of how far current MLLM agents can explore reliably in complex, open-ended urban environments.
Key facts
- UrbanGround is presented as the first sandbox that tests whether an MLLM agent's local perception of a street scene stays useful once the agent starts moving through a city.
- The sandbox is a physically constrained, real-scale replica of Hong Kong built from the city's territory-wide 3D geospatial data, with closed-loop first-person interaction and an interactive navigation map.
- The evaluation is structured around three research questions, moving from local scene grounding to navigation toward farther, less explicit destinations, to robustness under changing routes and pedestrian motion.
- Current MLLM agents show useful visual recognition and short-range spatial reasoning, but orientation and pedestrian-aware movement remain unreliable.
- Their central failure appears over extended exploration: local abilities fail to compose into sustained, goal-directed behavior, and errors accumulate without correction.
Why it matters
The paper opens on a distinction the authors treat as the crux of the problem: an MLLM can interpret a single street view well, but urban agency depends on whether that same local reading stays useful once the agent actually starts moving. UrbanGround is built to make that question testable, as a physically constrained replica of Hong Kong assembled from the city's actual territory-wide 3D geospatial data rather than a simplified or synthetic stand-in. Grounding the sandbox in a real city's geometry, instead of an idealized layout, is what lets the paper's findings speak to how these agents would behave in a genuinely real-world environment.
Who it affects
The direct audience is researchers building and evaluating MLLM-based agents for embodied navigation and spatial reasoning: UrbanGround gives them a real-scale, geospatially grounded city environment to test in. More broadly, it affects anyone trying to gauge how close current MLLM agents are to operating reliably in real urban space, since the paper's central finding, that useful short-range perception does not yet compose into reliable long-range navigation, describes a gap that any such deployment would need to close first.
How to use it
UrbanGround is a research sandbox, not a consumer product. Agents interact with it in a closed loop from a first-person view, entering the 3D city directly and using an interactive map to navigate; that is how the authors ran their three research questions. The text does not say whether the sandbox's code, environment or data will be made available to other researchers, so there is nothing to install or license yet; reproducing the setup would depend on the authors clarifying access.
How solid is it
The paper's grounding in real geospatial data, a physically constrained replica of Hong Kong built from territory-wide 3D data, is a substantive methodological choice, and the three-question structure gives the evaluation a clear progression from basic grounding to long-range, robustness-tested navigation. That said, the available text is descriptive rather than quantitative: it does not report accuracy figures, success rates or benchmark scores for how any agent actually performed, does not name which specific MLLM models were tested, and does not give the replica's numeric scale. It also carries no author names, institutional affiliations, publication venue or date, so there is no way from this text alone to check who did the work or where it will appear.
Risks and caveats
The paper's own headline finding is a limitation, not a capability: the agents' central failure shows up over extended exploration, where local abilities do not compose into sustained, goal-directed behavior and errors accumulate without correction, and orientation and pedestrian-aware movement remain unreliable even over shorter stretches. Because no specific models, scores or release plans appear in the available text, none of that can be checked or reproduced from this description alone; readers should treat UrbanGround as a newly proposed evaluation setup with an early, qualitative finding rather than an established benchmark with public results yet.
“Their central failure emerges over extended exploration, where local abilities do not compose into sustained goal-directed behavior and errors accumulate without effective correction.”
— the paper's authors