PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection

Forum topic · 小凯 · 2026-09-18

Summary

This arXiv paper (2609.19143) introduces PANORAMA, a vision-language model that unifies dense image captioning with pixel-level grounding. The authors identify that existing VLMs produce fluent captions but struggle to link referring phrases to accurate segmentation masks. They propose the task of panoptic grounded captioning, requiring models to describe foreground objects and background regions while grounding each phrase with pixel masks. Contributions include PanoCaps, a human-annotated benchmark built from panoptic segmentation datasets with near-complete pixel coverage, a phrase-mask matching protocol, and the generalized Panoptic Quality (gPQ) metric. PANORAMA reformulates phrase grounding as selecting masks from a pool of phrase-conditioned proposals generated by a pre-trained segmenter, trained jointly with caption generation. It achieves the best overall grounding performance on PanoCaps and matches or exceeds specialized models on multiple pixel-level grounding tasks. Authors: Sara Pieri, Evangelos Kazakos, Shizhe Chen, Josef Sivic, Cordelia Schmid. Code, data, and models are released on the project page.

Overview

Field: NLP / Vision-Language Models Authors: Sara Pieri, Evangelos Kazakos, Shizhe Chen, Josef Sivic, Cordelia Schmid Published: 2026-09-16 arXiv: 2609.19143

Abstract

Intelligent systems that act in the world require image understanding that is both comprehensive and spatially grounded. Current vision-language models (VLMs) can generate fluent and detailed image captions, but reliably associating them with image pixels remains challenging. Existing methods that combine dense captioning with pixel-level grounding often produce either incomplete descriptions or inaccurate segmentation masks. We study this problem through panoptic grounded captioning, a task that requires a VLM to describe both foreground objects and background regions while grounding each referring phrase with pixel-level masks. We make three contributions. First, we introduce PanoCaps, a human-annotated benchmark constructed from panoptic segmentation datasets. It provides dense captions with near-complete pixel coverage and entity-level text-image alignments, supporting both training and evaluation, along with a phrase-mask matching protocol and a generalized Panoptic Quality (gPQ) metric that jointly assesses textual and mask consistency. Second, we reformulate phrase grounding as selection from a pool of phrase-conditioned mask proposals. PANORAMA conditions a pre-trained segmenter on contextualized phrase representations to obtain candidate masks and learns to select the mask corresponding to each phrase. This interface is trained jointly with caption generation, enabling high-quality masks while allowing each phrase to refer to a single region or multiple instances. Third, PANORAMA achieves the best overall grounding performance on PanoCaps and matches or exceeds specialized models on multiple pixel-level grounding tasks, producing precise entity-level segmentation while maintaining detailed, mask-consistent captions. Code, data, and models are released on the project page.

Key Contributions

1. PanoCaps benchmark — human-annotated, built from panoptic segmentation datasets, with dense captions, entity-level alignments, phrase-mask matching protocol, and the gPQ metric. 2. PANORAMA model — phrase grounding as mask proposal selection from a phrase-conditioned pre-trained segmenter, jointly trained with caption generation. 3. State-of-the-art results — best overall grounding on PanoCaps; competitive with or superior to specialized pixel-grounding models.

--- *Auto-collected on 2026-09-18.*