This article is a Chinese deep-dive commentary on the paper PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception (Yana Wei, Hongbo Peng, Yanlin Lai, 2026, arXiv preprint), a work in computer vision / multimodal evaluation.
Key points
- The problem with 'impression scores': Popular metrics like CLIPScore embed AI-generated descriptions and reference captions and compare semantic similarity. A description can match 'in spirit' while omitting or corrupting critical facts (wrong counts, missing objects, blurred details), so models receive high scores for unreliable outputs. Semantic-embedding evaluation also fails to detect a systematic VLM failure mode: models identify elements but misdescribe relations between them.
- Why VQA alone isn't enough: Visual question answering is stricter but relies on human-authored answers that may be biased or incomplete, and it only measures what a model can answer when asked—not what it proactively notices in open-ended description.
- The PerceptionRubrics framework:
- 1,038 carefully selected information-dense images are annotated with golden captions produced through a circular peer-review consensus process: annotators independently describe, cross-review each other, and iterate until stable.
- Each golden caption is decomposed into instance-specific criteria—over 12,000 rubrics in total—split into two tracks:
- Must-Right: critical structural facts whose violation destroys the description's trustworthiness (e.g., 'the painting contains three figures').
- Easy-Wrong: fine-grained details that signal strong perception when correct (e.g., 'the woman wears a silver ring on her right hand').
- Gated scoring: if any Must-Right criterion is violated, the whole description receives a binary penalty (score drops to 0 or a very low value); otherwise the score equals Easy-Wrong accuracy. This 'one-vote veto' mimics human intuition better than linear averaging.
- The Reliability Gap: frontier open- and closed-source models recognize individual elements well yet fail on constraints and spatial relations among them—a systematic weakness rooted in training objectives that reward plausible-looking text rather than precise perception.
- Open–closed stratification: under strict rubric evaluation, a persistent ~8% perception deficit emerges for open-source models versus closed proprietary models, far larger than the 1–2 point gap seen in conventional benchmarks—likely reflecting differences in data scale, post-training alignment, and emphasis on perceptual precision.
- Human-aligned rigor: gated scoring correlates better with human judgments than linear averaging, since humans also treat key factual errors as disqualifying.
- The commentary frames the work philosophically: AI 'perception' today is largely symbolic pattern matching rather than genuine world understanding; rubric-based evaluation acts as a stress test exposing models that 'pretend to understand.'
- It situates PerceptionRubrics in the history of evaluation metrics—BLEU (2000s), METEOR/ROUGE (2010s), BERTScore/CLIPScore (2020s)—arguing stricter evaluation acts as a forcing function for capability improvements, echoing an evolution beyond the Turing test's 'fooling humans' standard toward reliability.
- It suggests implications for training: contrastive learning encourages 'overall matching' over 'detail precision,' motivating future fine-grained training objectives.
Experimental findings
Broader discussion in the article
References cited in the source
1. Wei, Y., Peng, H., & Lai, Y. (2026). *PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception*. arXiv preprint. 2. Papineni et al. (2002). BLEU. *ACL 2002*, 311–318. 3. Radford et al. (2021). Learning transferable visual models from natural language supervision. *ICML 2021*. 4. Turing, A. M. (1950). Computing machinery and intelligence. *Mind*, 59(236), 433–460. 5. Feynman, R. P. (1985). *Surely You're Joking, Mr. Feynman!* W. W. Norton & Company. 6. Natarajan, B. K. (1987). Machine learning: a theoretical approach. *STOC 1987*.