UrbanGround: A First-Person Sandbox Testing MLLM Spatial Agency in a Real-Scale 3D Replica of Hong Kong

Forum topic · 小凯 · 2026-08-29

Summary

UrbanGround (arXiv:2508.11373) is a benchmark sandbox built from territory-wide 3D geospatial data as a physically constrained replica of Hong Kong, designed to test how far multimodal large language model (MLLM) agents can convert local urban perception into reliable action at real city scale. The environment supports closed-loop first-person interaction and includes an interactive navigation map, letting agents enter and explore the 3D city directly. The authors—Tianjie Ju, Zheng Wu, and Yueqing Sun—structure their analysis around three research questions: whether agents can sufficiently ground local scenes after active observation to answer spatial questions; whether that grounding supports navigation as destinations become farther and more ambiguous; and whether resulting behavior stays robust under changing route availability and pedestrian movement. Findings show current MLLM agents possess useful atomic capabilities in visual recognition and short-range spatial reasoning, but direction awareness and pedestrian-aware motion remain unreliable. The core failure emerges in long-horizon exploration: local abilities fail to compose into sustained goal-directed behavior, with errors accumulating without effective correction. Released August 28, 2026.

论文概要

领域: Computer Vision Authors: Tianjie Ju, Zheng Wu, Yueqing Sun Published: 2026-08-28 arXiv: 2508.11373

Abstract (translated)

Multimodal large language models (MLLMs) can interpret a street view, but urban agency depends on whether such local evidence remains useful after the agent starts to move. This paper investigates how far current MLLM agents can turn local urban perception into reliable action in a complicated real-scale city.

The authors propose UrbanGround, the first sandbox that makes this question testable, built as a physically constrained replica of Hong Kong from territory-wide 3D geospatial data. UrbanGround supports closed-loop interaction from a first-person view and provides an interactive map for navigation, allowing agents to directly enter the 3D city and explore in first person.

The analysis follows the growth of the spatial problem through three research questions:

1. Local grounding: Can an agent, after actively observing a local scene, sufficiently ground it to answer spatial questions? 2. Navigation: Does this grounding support navigation as the destination becomes farther and more ambiguous? 3. Robustness: Is the resulting behavior stable under changes in route availability and pedestrian movement?

Findings

Current MLLM agents generally show useful atomic capabilities in visual recognition and short-range spatial reasoning.
Direction awareness and pedestrian-aware motion remain unreliable.
The core failure appears in long-horizon exploration: local capabilities cannot be composed into sustained goal-directed behavior, and errors accumulate without effective correction.