Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge (arXiv 2506.00002)

Forum topic · 小凯 · 2026-06-03

Summary

This paper (arXiv:2506.00002, posted 2026-06-03) by Seojeong Park, Jiho Choi, and Junyong Kang identifies a critical reliability weakness in multimodal large language models (MLLMs) used as automated evaluators, termed Perceptual Judgment Bias. When visual evidence conflicts with textual cues, MLLM judges tend to reward plausible narratives over perceptually correct answers. Controlled visual perturbation experiments show that existing multimodal judges frequently anchor on response text instead of their own visual perception, producing inconsistent, non-verifiable evaluations. To address this, the authors introduce the Perceptually Perturbed Judgment Dataset, built from minimally edited counterfactual responses that isolate perceptual errors and enable verifiable supervision. They further propose a unified training framework combining structured GRPO rewards with a batch-wise ranking objective, achieving coherent global ranking without explicit pairwise labels. Experiments across multiple MLLM-as-a-Judge benchmarks show significant improvements in perceptual fidelity, ranking consistency, and alignment with human judgments, establishing a scalable path toward perception-grounded, interpretable multimodal evaluators robust to visual-textual conflicts.

Paper Overview

Field: Computer Vision (CV) Authors: Seojeong Park, Jiho Choi, Junyong Kang Published: 2026-06-03 arXiv: 2506.00002

Abstract

Recent multimodal large language models have demonstrated strong reasoning ability, yet their reliability as automated evaluators remains limited by a critical weakness: when visual evidence conflicts with textual cues, MLLM judges tend to reward plausible narratives over perceptually correct answers.

The authors identify and systematically analyze this phenomenon, which they term Perceptual Judgment Bias. Through controlled visual perturbations, existing multimodal judges frequently anchor on the response text instead of their own visual perception, leading to inconsistent and non-verifiable evaluations.

Proposed Solution

Perceptually Perturbed Judgment Dataset: constructs minimally edited counterfactual responses that isolate perceptual errors and enable verifiable supervision.
Unified training framework: combines structured GRPO rewards with a batch-wise ranking objective, enabling coherent global ranking without explicit pairwise labels.

Results

Experiments on multiple MLLM-as-a-Judge benchmarks demonstrate that the method significantly improves:

Perceptual fidelity
Ranking consistency
Alignment with human evaluators

The results establish a scalable and generalizable path toward training perception-grounded, interpretable multimodal evaluators that are robust to visual-reasoning conflicts.

---

*Auto-collected on 2026-06-03.*