PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection

Forum topic · 小凯 · 2026-09-18

Summary

PANORAMA is a vision-language model for panoptic grounded captioning, a task requiring simultaneous dense captioning of foreground objects and background regions with pixel-level mask grounding for every referring phrase. The paper (arXiv:2609.19143) by Sara Pieri, Evangelos Kazakos, Shizhe Chen, Josef Sivic, and Cordelia Schmid introduces three contributions: (1) PanoCaps, a human-annotated benchmark built from panoptic segmentation datasets offering dense captions with near-complete pixel coverage and entity-level image-text alignment, plus a phrase-mask matching protocol and a generalized Panoptic Quality (gPQ) metric for joint text-mask evaluation; (2) a mask-selection interface that conditions a pretrained segmenter on contextual phrase representations to generate candidate masks and learns to select the mask matching each phrase, trained jointly with caption generation so phrases can refer to single regions or multiple instances; (3) state-of-the-art grounding performance on PanoCaps, matching or exceeding specialized pixel-grounding models while producing detailed, mask-consistent captions. Code, data, and models are released on the project page.

Paper Overview

Field: NLP / Vision-Language Models
Authors: Sara Pieri, Evangelos Kazakos, Shizhe Chen, Josef Sivic, Cordelia Schmid
Published: 2026-09-16
arXiv: 2609.19143

Abstract

Intelligent systems that act in the world require image understanding that is both comprehensive and spatially grounded. Current vision-language models (VLMs) can generate fluent and detailed image captions, but reliably associating them with image pixels remains challenging. Existing methods that combine dense captioning with pixel-level grounding often produce either incomplete descriptions or inaccurate segmentation masks.

The authors study this problem through panoptic grounded captioning, a task that requires a VLM to describe both foreground objects and background regions while grounding each referring phrase with pixel-level masks.

Key Contributions

1. PanoCaps benchmark: A human-annotated benchmark constructed from panoptic segmentation datasets. It provides dense captions with near-complete pixel coverage and entity-level image-text alignment for both training and evaluation, together with a phrase-mask matching protocol and a generalized Panoptic Quality (gPQ) metric that jointly evaluates text and mask consistency.

2. PANORAMA model: Phrase grounding is framed as selecting from a pool of phrase-conditioned mask proposals. A pretrained segmenter is conditioned on contextual phrase representations to obtain candidate masks, and the model learns to select the mask corresponding to each phrase. This interface is trained jointly with caption generation, allowing each phrase to refer to either a single region or multiple instances while generating high-quality masks.

3. Results: PANORAMA achieves the best overall grounding performance on PanoCaps and matches or exceeds specialized models on multiple pixel-level grounding tasks, producing precise entity-level segmentation while maintaining detailed, mask-consistent captions.

Code, data, and models are open-sourced on the project page.

---

*Auto-collected on 2026-09-18.*