Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptually Perturbed Training

Forum topic · 小凯 · 2026-06-03

Summary

This paper introduces Perceptual Judgment Bias, a critical weakness in multimodal large language model (MLLM) evaluators: when visual evidence conflicts with textual cues, MLLM judges tend to reward plausible narratives over perceptually correct answers. Through controlled visual perturbation experiments, the authors show that existing multimodal judges frequently anchor on response text rather than their own visual perception, producing inconsistent and non-verifiable evaluations. To address this, they present the Perceptually Perturbed Judgment Dataset, built from minimally edited counterfactual responses that isolate perceptual errors and enable verifiable supervision. Using this dataset, they develop a unified training framework combining structured GRPO rewards with a batch-wise ranking objective, achieving coherent global ranking without explicit pairwise labels. Experiments across multiple MLLM-as-a-Judge benchmarks demonstrate significant improvements in perceptual fidelity, ranking consistency, and alignment with human evaluation. The work (arXiv:2506.00002) offers a scalable path toward training perception-grounded, interpretable multimodal evaluators robust to visual-reasoning conflicts.

Paper Overview

Field: Computer Vision Authors: Seojeong Park, Jiho Choi, Junyong Kang Published: 2026-06-03 arXiv: 2506.00002

Abstract

Recent multimodal large language models have demonstrated strong reasoning ability, yet their reliability as automated evaluators remains limited by a critical weakness: when visual evidence conflicts with textual cues, MLLM judges tend to reward plausible narratives over perceptually correct answers. The authors identify and systematically analyze this phenomenon, which they term Perceptual Judgment Bias.

Through controlled visual perturbations, they find that existing multimodal judges frequently anchor on the response text instead of their own visual perception, leading to inconsistent and non-verifiable evaluations.

Method

To address this issue, the authors introduce the Perceptually Perturbed Judgment Dataset, which constructs minimally edited counterfactual responses that isolate perceptual errors and enable verifiable supervision. Based on this dataset, they develop a unified training framework combining:

Structured GRPO rewards
A batch-wise ranking objective that achieves coherent global ranking without explicit pairwise labels

Results

Experiments across multiple MLLM-as-a-Judge benchmarks show that the proposed method significantly improves:

Perceptual fidelity
Ranking consistency
Alignment with human evaluators

The results establish a scalable and generalizable path for training perception-grounded, interpretable multimodal evaluators that are robust to visual-reasoning conflicts.

--- *Auto-collected on 2026-06-03*