Paper Overview
Field: Computer Vision Authors: Song Tang, Shuming Hu, Xincheng Shuai Posted: 2026-07-04 arXiv: 2507.03235
Summary
Existing referring segmentation models passively process static images captured from fixed perspectives, which limits their applicability to Embodied AI, where agents must perform active perception in continuous 360° environments. To bridge this gap, the paper introduces a new task: Active Panoramic Referring Segmentation (APRS). In this setting, an agent must adjust its viewing direction (Δθ, Δφ) to explore the 360° environment, seek the target object specified by a user instruction, and segment it.
To address this challenge, the authors propose PanoSeeker, a memory-augmented APRS agent. Instead of relying on heuristic scanning, PanoSeeker integrates a Vision-Language Model (VLM) with EgoSphere, an explicit spatial visual memory. By progressively integrating sequential local observations into a unified panoramic representation, EgoSphere enables the agent to plan efficient and non-redundant search trajectories. Once the target is found, the agent performs active viewpoint alignment and outputs the segmentation mask.
Additionally, the authors curate a dataset of expert-annotated search trajectories with memory timelines for supervised fine-tuning (SFT), followed by reinforcement learning post-training to explicitly optimize exploration efficiency. Extensive experiments on the newly established APRS benchmark show that PanoSeeker significantly outperforms adapted state-of-the-art baselines in both search efficiency and segmentation accuracy.
Original Abstract
> Existing referring segmentation models passively process static images captured from fixed perspectives, limiting their applicability in Embodied AI, where agents must perform active perception in the continuous 360° environments. To bridge this gap, we introduce a novel task: Active Panoramic Referring Segmentation (APRS). In this setting, an agent is required to adjust its viewing direction (Δθ, Δφ) to explore the 360° environment, seeking the object specified by a user instruction for segmentation. To tackle this challenging task, we propose PanoSeeker, a memory-augmented agent for efficient APRS. Rather than relying on heuristic scanning, PanoSeeker integrates a Vision-Language Model (VLM) with EgoSphere, an explicit spatial visual memory. By progressively integrating sequential local observations into a unified 360° representation, EgoSphere enables the agent to plan efficient and non-redundant search trajectories. Once the target is found, the agent performs active viewpoint alig...
---
*Auto-collected on 2026-07-05*