Seek to Segment: Active Perception for Panoramic Referring Segmentation (PanoSeeker)

Forum topic · 小凯 · 2026-07-06

Summary

This paper introduces Active Panoramic Referring Segmentation (APRS), a new task bridging the gap between passive referring segmentation models and embodied AI agents operating in continuous 360-degree environments. In APRS, an agent must actively adjust its viewing direction (Δθ, Δφ) to explore a panoramic environment and locate the object specified by a user instruction before producing a segmentation mask. The authors propose PanoSeeker, a memory-augmented agent that combines a Vision-Language Model (VLM) with EgoSphere, an explicit spatial visual memory that progressively integrates sequential local observations into a unified 360-degree representation, enabling efficient, non-redundant search trajectories instead of heuristic scanning. The method is trained via supervised fine-tuning on a curated expert-annotated search-trajectory dataset with memory timelines, followed by reinforcement learning post-training to optimize exploration efficiency. Experiments on a newly established APRS benchmark show that PanoSeeker outperforms adapted state-of-the-art baselines in both search efficiency and segmentation accuracy. Paper: arXiv 2607.02497 (cs.CV), by Song Tang, Shuming Hu, Xincheng Shuai, Henghui Ding, and Yu-Gang Jiang.

Paper Overview

Field: Computer Vision
Authors: Song Tang, Shuming Hu, Xincheng Shuai, Henghui Ding, Yu-Gang Jiang
Published: 2026-07-02
arXiv: 2607.02497
Category: cs.CV

Abstract

Existing referring segmentation models passively process static images captured from fixed perspectives, limiting their applicability in Embodied AI, where agents must perform active perception in continuous 360° environments. To bridge this gap, the authors introduce a novel task: Active Panoramic Referring Segmentation (APRS). In this setting, an agent adjusts its viewing direction (Δθ, Δφ) to explore the 360° environment, seeking the object specified by a user instruction for segmentation.

Method: PanoSeeker

To tackle this challenging task, the authors propose PanoSeeker, a memory-augmented agent for efficient APRS:

Rather than relying on heuristic scanning, PanoSeeker integrates a Vision-Language Model (VLM) with EgoSphere, an explicit spatial visual memory.
EgoSphere progressively integrates sequential local observations into a unified 360° representation, enabling the agent to plan efficient and non-redundant search trajectories.
Once the target is found, the agent performs active viewpoint alignment and outputs the segmentation mask.

Training

A curated expert-annotated search-trajectory dataset with memory timelines is used for supervised fine-tuning.
Reinforcement learning post-training then explicitly optimizes PanoSeeker's exploration efficiency.

Results

Extensive experiments on a newly established APRS benchmark show that PanoSeeker achieves superior performance in both search efficiency and segmentation accuracy, significantly outperforming adapted state-of-the-art baseline methods.

---

*Auto-collected on 2026-07-06*