PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception

Forum topic · 小凯 · 2026-06-30

Summary

PerceptionRubrics is a rubric-based evaluation framework for multimodal models that bridges the gap between saturated benchmark scores and real-world brittleness. The framework pairs 1,038 information-dense images with more than 12,000 instance-specific rubrics, derived from golden captions built through a Circular Peer-Review consensus pipeline. Rubrics are distilled into a dual-stream system: Must-Right criteria covering essential facts, and Easy-Wrong criteria targeting fine-grained details. A key innovation is the Gated Scoring mechanism, which applies sharp binary penalties when a model fails mandatory visual facts, unlike conventional linear averaging. Extensive evaluation reveals three findings: a reliability gap where models verify fragmentary elements correctly but collapse under strict conjunctive constraints; a persistent ~8% perception gap between open-source and proprietary frontier models despite converging reasoning trends; and stronger alignment with human judgment compared to traditional benchmarks, validating that strict perceptual fidelity is a prerequisite for reliable generation. Paper: arXiv 2606.28322.

Paper Overview

Field: Computer Vision Authors: Yana Wei, Hongbo Peng, Yanlin Lai Published: 2026-06-26 arXiv: 2606.28322

Abstract

We introduce PerceptionRubrics, a rubric-based evaluation framework that addresses the gap between saturated benchmark scores and real-world brittleness. Shifting evaluation from holistic semantic matching to rigorous atomic auditing, PerceptionRubrics pairs 1,038 information-dense images with over 12,000 instance-specific rubrics. These criteria are derived from golden captions constructed via a novel Circular Peer-Review consensus pipeline and then distilled into a dual-stream system of Must-Right (essential facts) and Easy-Wrong (fine-grained details) rubrics. Crucially, PerceptionRubrics implements a Gated Scoring mechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties.

Key Findings

The Reliability Gap: Models typically verify fragmentary elements correctly but fail under strict conjunctive constraints, exposing brittleness in dense visual domains.
Open-Source vs. Closed-Source Stratification: Contrary to converging reasoning trends, the study reveals a persistent ~8% perception gap between open-source and proprietary frontier models.
Human-Aligned Rigor: The gated metric aligns better with human judgment than traditional benchmarks, validating that strict perceptual fidelity is a prerequisite for reliable generation.

Method Highlights

1,038 information-dense images paired with 12,000+ instance-specific rubrics.
Golden captions produced via a Circular Peer-Review consensus pipeline.
Dual-stream rubric system: Must-Right (essential facts) and Easy-Wrong (fine-grained details).
Gated Scoring: mandatory visual-fact failures trigger sharp binary penalties rather than degrading averages.

---

Paper link: https://arxiv.org/abs/2606.28322