Paper Overview
Field: Computer Vision Authors: Song Tang, Shuming Hu, Xincheng Shuai arXiv: 2507.03235
Summary
Existing referring segmentation models passively process static images captured from fixed perspectives, limiting their applicability in Embodied AI, where agents must perform active perception in continuous 360° environments. To bridge this gap, the paper introduces a novel task: Active Panoramic Referring Segmentation (APRS). In this setting, an agent must adjust its viewing direction (Δθ, Δφ) to explore a 360° environment, seeking the object specified by a user instruction and segmenting it.
To tackle this challenging task, the authors propose PanoSeeker, a memory-augmented agent for efficient APRS. Rather than relying on heuristic scanning, PanoSeeker integrates a Vision-Language Model (VLM) with EgoSphere, an explicit spatial visual memory. By progressively integrating sequential local observations into a unified 360° representation, EgoSphere enables the agent to plan efficient and non-redundant search trajectories. Once the target is found, the agent performs active viewpoint alignment and outputs the segmentation mask.
Additionally, the authors curate a dataset of expert-annotated search trajectories with memory timelines for supervised fine-tuning (SFT), followed by reinforcement learning post-training to explicitly optimize PanoSeeker's exploration efficiency. Extensive experiments on the newly established APRS benchmark show that PanoSeeker significantly outperforms adapted state-of-the-art baselines in both search efficiency and segmentation accuracy.
Key Contributions
- New task (APRS): extends referring segmentation to active, egocentric search in 360° environments.
- PanoSeeker agent: combines a VLM with EgoSphere explicit spatial memory for planning efficient search trajectories.
- Training pipeline: expert-annotated trajectory dataset with memory timelines for SFT, plus reinforcement learning post-training for exploration efficiency.
- Benchmark: a new APRS benchmark demonstrating substantial gains over adapted state-of-the-art baselines.
*Auto-collected on 2026-07-05.*