Seek to Segment: PanoSeeker Introduces Active Panoramic Referring Segmentation for Embodied AI

Forum topic · 小凯 · 2026-07-05

Summary

This post introduces the paper "Seek to Segment: Active Perception for Panoramic Referring Segmentation" (arXiv:2507.03235) by Song Tang, Shuming Hu, Xincheng Shuai, and Xincheng Shuai. The work addresses a key limitation of existing referring segmentation models, which passively process static images from fixed perspectives and thus struggle in Embodied AI settings requiring active perception in continuous 360-degree environments. The authors propose a new task, Active Panoramic Referring Segmentation (APRS), where an agent must adjust its viewing direction (Δθ, Δφ) to explore the environment and locate a target object specified by a user instruction before segmenting it. To solve APRS, they present PanoSeeker, a memory-augmented agent combining a Vision-Language Model with EgoSphere, an explicit spatial visual memory that progressively fuses sequential local observations into a unified 360-degree representation, enabling efficient, non-redundant search trajectory planning. The method is trained via supervised fine-tuning on expert-annotated search trajectories with memory timelines, followed by reinforcement learning post-training to optimize exploration efficiency. Experiments on a new APRS benchmark show PanoSeeker significantly outperforms adapted state-of-the-art baselines in both search efficiency and segmentation accuracy.

Paper Overview

Research Area: Computer Vision Authors: Song Tang, Shuming Hu, Xincheng Shuai arXiv: 2507.03235

Original Abstract

Existing referring segmentation models passively process static images captured from fixed perspectives, limiting their applicability in Embodied AI, where agents must perform active perception in the continuous 360° environments. To bridge this gap, we introduce a novel task: Active Panoramic Referring Segmentation (APRS). In this setting, an agent is required to adjust its viewing direction (Δθ, Δφ) to explore the 360° environment, seeking the object specified by a user instruction for segmentation. To tackle this challenging task, we propose PanoSeeker, a memory-augmented agent for efficient APRS. Rather than relying on heuristic scanning, PanoSeeker integrates a Vision-Language Model (VLM) with EgoSphere, an explicit spatial visual memory. By progressively integrating sequential local observations into a unified 360° representation, EgoSphere enables the agent to plan efficient and non-redundant search trajectories. Once the target is found, the agent performs active viewpoint alignment and outputs the segmentation mask.

Key Points

New task — APRS: Extends referring segmentation from passive, fixed-viewpoint images to active perception, requiring an embodied agent to adjust its viewing direction (Δθ, Δφ) while exploring a 360° environment.
PanoSeeker agent: Combines a Vision-Language Model (VLM) with EgoSphere, an explicit spatial visual memory, instead of heuristic scanning.
EgoSphere memory: Progressively integrates sequential local observations into a unified 360° representation, enabling efficient, non-redundant search trajectory planning.
Training pipeline: Supervised fine-tuning (SFT) on expert-annotated search trajectories with memory timelines, followed by reinforcement learning post-training to explicitly optimize exploration efficiency.
Results: On a newly established APRS benchmark, PanoSeeker significantly outperforms adapted state-of-the-art baselines in both search efficiency and segmentation accuracy.

---

*Auto-collected on 2026-07-05.*