UrbanGround: Testing MLLM Agents' Spatial Agency in a Real-Scale 3D Replica of Hong Kong

Forum topic · 小凯 · 2026-08-30

Summary

UrbanGround is a physically constrained sandbox environment built from territory-wide 3D geospatial data of Hong Kong, designed to test whether multimodal large language model (MLLM) agents can convert local urban perception into reliable action at real city scale. The environment supports closed-loop first-person interaction and includes an interactive navigation map, allowing agents to enter and explore the 3D city directly. The paper structures its analysis around three research questions of increasing difficulty: whether agents can understand local scenes well enough to answer spatial questions via active observation; whether that understanding supports navigation to increasingly distant and ambiguous destinations; and whether behaviors remain robust under changes in route availability and pedestrian movement. Findings show that current MLLM agents possess useful atomic abilities in visual recognition and short-range spatial reasoning, but orientation and pedestrian-aware movement remain unreliable. The core failure emerges during long-horizon exploration: local capabilities fail to compose into sustained goal-directed behavior, and errors accumulate without effective correction. The paper was posted on arXiv (2608.27456) in the computer vision field.

论文概要 (Paper Overview)

Research Area: Computer Vision (CV)

Authors: Tianjie Ju, Zheng Wu, Yueqing Sun, Yuhan Cui, Bobo Li, Shengqiong Wu, Pengzhou Cheng, Haodong Zhao, Zongru Wu, Xinbei Ma, Doris Zhang, Kunling Li, Mong-Li Lee, Wynne Hsu, Hao Fei, Qi Gu, Gongshen Liu, Zhuosheng Zhang

arXiv: 2608.27456

Abstract

Multimodal large language models (MLLMs) can interpret a street view, but urban agency depends on whether such local evidence remains useful after the agent starts to move. In this paper, the authors investigate how far current MLLM agents can turn local urban perception into reliable action in a complicated real-scale city. They propose UrbanGround, the first sandbox to make this question testable in a physically constrained replica of Hong Kong built from territory-wide 3D geospatial data.

UrbanGround supports closed-loop interaction from a first-person view and provides an interactive map for navigation. Agents can directly enter the 3D city and explore from a first-person perspective.

Research Questions

The analysis follows the growth of the spatial problem through three research questions:

1. Local scene understanding — Can an agent understand a local scene sufficiently through active observation to answer spatial questions? 2. Navigation — Does this understanding support navigation as destinations become farther and more ambiguous? 3. Behavioral robustness — Do behaviors remain valid when route availability and pedestrian movement change?

Findings

Contemporary MLLM agents show useful atomic abilities in visual recognition and short-range spatial reasoning.
Orientation and pedestrian-aware movement remain unreliable.
The core failure appears in long-horizon exploration: local capabilities do not compose into sustained goal-directed behavior, and errors accumulate without effective correction.

---

*Auto-collected on 2026-08-30. Originally posted in Chinese on zhichai.net.*