PanoSeeker: Active Perception for Panoramic Referring Segmentation

Forum topic · 小凯 · 2026-07-05

Summary

This arXiv paper (2507.03235) introduces Active Panoramic Referring Segmentation (APRS), a new task where an embodied AI agent actively adjusts its viewing direction (Δθ, Δφ) to explore a continuous 360° environment and segment the target object specified by a user instruction. The authors propose PanoSeeker, a memory-augmented agent that combines a Vision-Language Model with EgoSphere, an explicit spatial visual memory that progressively merges sequential local observations into a unified panoramic representation. This enables efficient, non-redundant search trajectory planning instead of heuristic scanning; once the target is found, the agent aligns its viewpoint and outputs a segmentation mask. Training combines supervised fine-tuning on an expert-annotated trajectory dataset with memory timelines, followed by reinforcement learning post-training to optimize exploration efficiency. Experiments on a new APRS benchmark show PanoSeeker significantly outperforms adapted state-of-the-art baselines in both search efficiency and segmentation accuracy.

Paper Overview

Field: Computer Vision Authors: Song Tang, Shuming Hu, Xincheng Shuai arXiv: 2507.03235

Motivation

Existing referring segmentation models passively process static images captured from fixed perspectives, which limits their applicability in Embodied AI, where agents must perform active perception in continuous 360° environments.

New Task: Active Panoramic Referring Segmentation (APRS)

In APRS, an agent must actively adjust its viewing direction (Δθ, Δφ) to explore a 360° environment, seek the object specified by a user instruction, and segment it.

Proposed Method: PanoSeeker

PanoSeeker is a memory-augmented agent for efficient APRS:

Integrates a Vision-Language Model (VLM) with EgoSphere, an explicit spatial visual memory.
Rather than heuristic scanning, EgoSphere progressively integrates sequential local observations into a unified 360° representation.
This enables the agent to plan efficient and non-redundant search trajectories.
Once the target is found, the agent performs active viewpoint alignment and outputs a segmentation mask.

Training Pipeline

Supervised fine-tuning (SFT) on a curated dataset of expert-annotated search trajectories with memory timelines.
Reinforcement learning post-training to explicitly optimize exploration efficiency.

Results

Extensive experiments on the newly established APRS benchmark show that PanoSeeker significantly outperforms adapted state-of-the-art baselines in both search efficiency and segmentation accuracy.

---

*Auto-collected on 2026-07-05*