PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception

Forum topic · 小凯 · 2026-06-30

Summary

PerceptionRubrics is a rubric-based evaluation framework for multimodal models that addresses the gap between saturated benchmark scores and real-world brittleness. The framework shifts evaluation from holistic semantic matching to rigorous atomic auditing, pairing 1,038 information-dense images with over 12,000 instance-specific rubrics. Rubrics are derived from golden captions built via a Circular Peer-Review consensus pipeline and distilled into a dual-stream system of Must-Right (essential facts) and Easy-Wrong (fine-grained details) criteria. A Gated Scoring mechanism applies sharp binary penalties for failures on mandatory visual facts instead of linear averaging. Evaluations reveal three key findings: a reliability gap where models verify individual elements but fail under conjunctive constraints; a persistent ~8% perception gap between open-source and proprietary frontier models; and stronger alignment with human judgment than traditional benchmarks, validating that strict perceptual fidelity underpins reliable generation.

Paper Overview

Field: Computer Vision (CV) Authors: Yana Wei, Hongbo Peng, Yanlin Lai Date: 2026-06-26 arXiv: 2606.28322

Abstract

We introduce PerceptionRubrics, a rubric-based evaluation framework that addresses the gap between saturated benchmark scores and real-world brittleness. Shifting evaluation from holistic semantic matching to rigorous atomic auditing, PerceptionRubrics pairs 1,038 information-dense images with over 12,000 instance-specific rubrics. These criteria are derived from golden captions constructed via a novel Circular Peer-Review consensus pipeline and then distilled into a dual-stream system of Must-Right (essential facts) and Easy-Wrong (fine-grained details) rubrics. Crucially, PerceptionRubrics implements a Gated Scoring mechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties.

Key Findings

The Reliability Gap: Models often correctly verify individual fragment elements but fail under strict conjunctive constraints, exposing brittleness in dense domains.
Open-source vs. Proprietary Stratification: Contrary to reasoning trends, a persistent ~8% perception gap exists between open-source and proprietary frontier models.
Human-Aligned Rigor: The gated metric aligns more closely with human judgment than traditional benchmarks, validating that strict perceptual fidelity is a prerequisite for reliable generation.

Methodology Highlights

1,038 information-dense images paired with 12,000+ instance-specific rubrics.
Rubrics derived from golden captions via a Circular Peer-Review consensus pipeline.
Dual-stream rubric design: Must-Right (essential facts) vs. Easy-Wrong (fine-grained details).
Gated Scoring: binary penalties for failures on mandatory visual facts, rather than linear averaging.

---

*Auto-collected on 2026-06-30.*