Inside the Unfair Judge: Mechanistic Interpretability of LLM-as-Judge Scoring Bias (arXiv 2607.11871)

Forum topic · 小凯 · 2026-07-15

Summary

A new paper (arXiv 2607.11871, Zixiang Xu et al.) argues that LLM-as-judge scoring biases can be explained at the representation level, not just the input-output level. Across seven judge models, seven bias types, and nine benchmarks, the authors report three findings. Geometry: biased inputs are displaced from a tight activation manifold along a low-dimensional, type-specific subspace that sharpens with model depth and is consistently recovered by three estimator families. Causal control: steering hidden states along this subspace bidirectionally drives scoring—forward shifts reproduce biased scores on clean inputs, reverse shifts restore baseline scores on biased inputs—while matched random directions produce effects an order of magnitude smaller. Operationally, a simple linear probe on these bias-direction features predicts judge failures on three fully unseen benchmarks, substantially outperforming text-based alternatives. The work unifies geometry, causal control, and prediction in one framework.

Paper Overview

Field: Machine Learning Authors: Zixiang Xu, Sixian Li, Huaxing Liu, Xiang Wang, Shuai Li Published: 2026-07-13 arXiv: 2607.11871

Abstract

Existing studies of LLM-as-judge scoring bias work predominantly at the input-output level: they perturb inputs, measure score deltas, and propose prompt-level mitigations. This paper argues that the same biases admit a representation-level account in the judge's hidden state—complementary to the input-output view and operationally more useful. Three findings are reported across seven judge models, seven bias types, and nine benchmarks.

1. Geometry

Baseline judging inputs occupy a tight activation manifold, while biased inputs are displaced along a low-dimensional, type-specific subspace. This subspace sharpens with depth and is recovered consistently by three families of estimators.

2. Causal Control

Steering hidden states along this subspace drives scoring in both directions:

Forward shifts reproduce biased scoring on clean inputs.
Reverse shifts restore baseline scoring on biased inputs.
Matched random directions produce shifts an order of magnitude smaller.

3. Operational Value

A simple linear projection onto these bias-direction features can predict judge failures on three fully unseen benchmarks, significantly outperforming text-based alternatives.

Reading bias as activation geometry rather than input-output noise unifies geometric structure, causal control, and operational prediction within a single framework.

Link: https://arxiv.org/abs/2607.11871

--- *Auto-collected 2026-07-15*