Beyond Scores: What Happens Inside an LLM Judge's Brain When It Scores Summaries
*Daily paper recommendation - September 3, 2026. Original forum post in Chinese, translated and condensed.*
The Paper
Original title: Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation Authors: Himil Vasava, Ming Jiang Categories: cs.CL, cs.LG
LLM-as-a-Judge systems now score AI-generated text everywhere—from paper review to training signal generation. But when a large language model outputs "7.5/10" for a summary, what actually happens inside? This paper uses mechanistic interpretability as an X-ray machine to peer into the judge's internals.
Why This Matters
Traditional summarization evaluation is either expensive (human raters) or shallow (ROUGE only counts lexical overlap and misses semantics). LLM judges promise semantic understanding at scale—but their decision-making is a black box. If the judge is opaque, we cannot trust its scores, nor the models trained on its signals.
Methods
The authors study two judge models—Themis (Llama-3-8B based) and Prometheus (Mistral-7B based)—plus untuned Llama-3-8B as a control, using:
- Eight perturbation attacks on summaries, each with clean/contaminated pairs: lexical substitution, grammar corruption, information omission, hallucinated additions, redundant padding, scrambled event order, ambiguous coreference, and style drift. These target the two core NLG quality dimensions: readability and adequacy.
- Logit lens: projecting intermediate layers to see the model's evolving "internal monologue."
- Causal tracing and attention head knockout: disabling components to test causal responsibility.
- Some attention heads specialize in readability detection; others in adequacy detection; others act as information relays.
- Critical heads near the crystallization layer (25–26) cause steep performance drops when knocked out, while early-layer knockouts have minor effects thanks to redundant pathways.
- Some heads appear to perform safety checks: removing them makes the judge vulnerable to subtle adversarial substitutions—a potential attack surface.
- Evaluation ability has concrete, interpretable structure—it can be understood, improved, and repaired.
- Bias is locatable: style sensitivity, flawed MLP integration, or premature crystallization can be pinpointed to specific circuits.
- Safer systems: mechanistic understanding enables detecting adversarial manipulation and designing multi-judge architectures.
- Vasava, H., & Jiang, M. (2026). Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation. arXiv preprint.
- Meng, Y., et al. (2022). Locating and Editing Factual Associations in GPT. NeurIPS.
- Nanda, N., et al. (2023). Transformer Circuits Thread. Anthropic.
- Zou, A., et al. (2023). Representation Engineering: A Top-Down Approach to AI Transparency. arXiv preprint.
Key Findings
A two-stage pipeline
1. Sensory scanning (layers 1–15): Attention heads compare local features between the summary and reference text, routing results to the final sequence position where the score is produced. MLP contributions are suppressed here. 2. Integrated judgment (layers 15+): MLP layers dominate, aggregating signals like a panel of small judges. The decision crystallizes at a specific layer (layer 26 for Themis, layer 25 for Prometheus)—perturbations after this point no longer change the score.
Fine-tuning sculpts rather than builds
The untuned base model already exhibits a similar pipeline, but with less stage separation (early MLPs also feed the final decision) and later crystallization. Fine-tuning suppresses early-MLP participation and moves decision crystallization earlier—sculpting an existing structure rather than constructing one from scratch.
Specialized heads and vulnerabilities
The logit lens view
Early layers produce chaotic token predictions; mid-layers surface evaluation words ("good," "clear," "accurate"); near the crystallization layer, numeric tokens spike; after it, the score is effectively fixed.
Implications
Takeaway
The paper marks a shift from blind trust to *informed trust* in LLM judges. As the original post puts it: a judge we don't understand is itself unfair to those being judged.