ESI-Bench: A Benchmark for Embodied Spatial Intelligence that Closes the Perception-Action Loop

Forum topic · 小凯 · 2026-05-20

Summary

ESI-BENCH (arXiv:2505.14305) is a comprehensive benchmark for embodied spatial intelligence, spanning 10 task categories and 29 subcategories, built on OmniGibson and grounded in Spelke's core knowledge systems. Unlike prior spatial intelligence formulations that assume oracle observations, ESI-BENCH recasts the observer as an actor: agents must decide which abilities to deploy—perception, locomotion, and manipulation—and how to sequence them to actively gather task-relevant evidence. Experiments on state-of-the-art multimodal LLMs show that active exploration substantially outperforms passive counterparts, that agents spontaneously discover emergent spatial strategies without explicit instruction, and that random multi-view captures often add noise rather than signal. Most failures stem not from weak perception but from action blindness—poor action choices yield poor observations and cascading errors. Explicit 3D grounding stabilizes depth-sensitive reasoning, but imperfect 3D representations can be more harmful than 2D baselines. Human studies reveal a metacognitive gap: models commit prematurely with high confidence regardless of evidence quality, unlike humans who seek falsifying viewpoints and revise beliefs.

Overview

Field: NLP Authors: Yining Hong, Jiageng Liu, Han Yin Published: 2026-05-19 arXiv: 2505.14305

Abstract (translated)

Spatial intelligence unfolds through a perception-action loop: agents act to acquire observations, and reason about how observations vary as a function of action. Rather than passively processing what is seen, they actively uncover what is unseen — occluded structure, dynamics, containment, and functionality that cannot be resolved from passive sensing alone. The authors move beyond prior formulations of spatial intelligence that assume oracle observations by recasting the observer as an actor.

They introduce ESI-BENCH, a comprehensive benchmark for embodied spatial intelligence spanning 10 task categories and 29 subcategories, built on OmniGibson and grounded in Spelke's core knowledge systems. Agents must decide what abilities to deploy — perception, locomotion, and manipulation — and how to sequence these abilities to actively accumulate task-relevant evidence.

Key Findings

Extensive experiments on state-of-the-art MLLMs show that active exploration significantly outperforms passive counterparts.
Agents spontaneously discover emergent spatial strategies without explicit instruction.
Random multi-view capture, despite consuming more images, often adds noise rather than signal.
Most failures stem not from weak perception but from action blindness: poor action choices lead to poor observations, triggering cascading errors.
Explicit 3D grounding stabilizes reasoning on depth-sensitive tasks, but imperfect 3D representations are more harmful than 2D baselines because they distort spatial relations.
Human studies reveal a metacognitive gap: unlike humans, who seek falsifying viewpoints and revise beliefs under contradiction, models commit prematurely with high confidence regardless of evidence quality — a gap that neither better perception nor more embodied interaction alone can bridge.