Seek to Segment: Active Perception for Panoramic Referring Segmentation (PanoSeeker)

Forum topic · 小凯 · 2026-07-06

Summary

PanoSeeker addresses Active Panoramic Referring Segmentation (APRS), a new embodied AI task where an agent must actively adjust its viewing direction (Δθ, Δφ) to explore a continuous 360° environment and find the object specified by a user instruction for segmentation. Unlike existing referring segmentation models that passively process static images from fixed perspectives, PanoSeeker combines a Vision-Language Model (VLM) with EgoSphere, an explicit spatial visual memory that progressively merges sequential local observations into a unified 360° representation. This enables the agent to plan efficient, non-redundant search trajectories instead of heuristic scanning. Once the target is located, the agent performs active viewpoint alignment and outputs a segmentation mask. The authors curate an expert-annotated dataset of search trajectories with memory timelines for supervised fine-tuning, followed by reinforcement learning post-training to optimize exploration efficiency. Experiments on a newly established APRS benchmark show PanoSeeker outperforms adapted state-of-the-art baselines in both search efficiency and segmentation accuracy. Paper: arXiv 2607.02497 by Song Tang, Shuming Hu, Xincheng Shuai, Henghui Ding, and Yu-Gang Jiang.

Overview

Field: Computer Vision
Authors: Song Tang, Shuming Hu, Xincheng Shuai, Henghui Ding, Yu-Gang Jiang
Published: 2026-07-02
arXiv: 2607.02497
Category: cs.CV

Key Points

Introduces Active Panoramic Referring Segmentation (APRS), a new task for embodied AI in which an agent adjusts its viewing direction $(\Delta\theta, \Delta\phi)$ to explore a continuous 360° environment and seek the target object specified by a user instruction for segmentation.
Existing referring segmentation models passively process static images from fixed perspectives, limiting their applicability to embodied settings requiring active perception.
Proposes PanoSeeker, a memory-augmented agent that integrates a Vision-Language Model (VLM) with EgoSphere, an explicit spatial visual memory that progressively integrates sequential local observations into a unified 360° representation.
EgoSphere enables planning of efficient, non-redundant search trajectories instead of heuristic scanning. Once the target is found, the agent performs active viewpoint alignment and outputs a segmentation mask.
The authors curate an expert-annotated dataset of search trajectories with memory timelines for supervised fine-tuning, followed by reinforcement learning post-training to explicitly optimize exploration efficiency.
Extensive experiments on a newly established APRS benchmark show PanoSeeker achieves superior search efficiency and segmentation accuracy, significantly outperforming adapted state-of-the-art baselines.

Original Abstract (excerpt)

> Existing referring segmentation models passively process static images captured from fixed perspectives, limiting their applicability in Embodied AI, where agents must perform active perception in the continuous 360 $^\circ$ environments. To bridge this gap, we introduce a novel task: Active Panoramic Referring Segmentation (APRS). In this setting, an agent is required to adjust its viewing direction ( $\Delta\theta, \Delta\phi$ ) to explore the 360 $^\circ$ environment, seeking the object specified by a user instruction for segmentation. To tackle this challenging task, we propose PanoSeeker, a memory-augmented agent for efficient APRS. Rather than relying on heuristic scanning, PanoSeeker integrates a Vision-Language Model (VLM) with EgoSphere, an explicit spatial visual memory...

*Auto-collected on 2026-07-06.*