Beyond Scores: What Happens Inside an LLM's Brain When It Acts as a Judge

Forum topic · 小凯 · 2026-09-02

Summary

A Chinese tech forum post reviews the paper 'Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation' by Himil Vasava and Ming Jiang. Using mechanistic interpretability techniques—logit lens, causal tracing, and attention head knockout—the authors probe how judge models Themis (Llama-3-8B based) and Prometheus (Mistral-7B based) score summaries on readability and adequacy. Key findings: LLM judges operate as a two-stage pipeline; attention heads in the first ~15 layers perform local comparisons between candidate and reference texts while MLP contributions are suppressed, then MLPs dominate above layer 15 to integrate signals, with the score 'crystallizing' around layer 26 (Themis) or 25 (Prometheus). Untuned Llama-3-8B shows a similar but less separated structure with later crystallization, suggesting fine-tuning sculpts innate capability rather than building it from scratch. Head-knockout experiments reveal specialized heads for readability versus adequacy detection and critical heads near the crystallization layer whose removal collapses judging ability, exposing adversarial vulnerabilities. The post argues that interpretability enables locating and fixing bias, designing safer multi-judge systems, and moving from blind to informed trust in AI evaluation.

X-Ray on the Judge's Bench: What Is an AI Thinking When It Learns to 'Score'?

A daily paper recommendation post (September 3, 2026) from zhichai.net reviewing:

Paper: *Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation* Authors: Himil Vasava, Ming Jiang Categories: cs.CL, cs.LG

Background

LLM-as-a-Judge has become ubiquitous: large language models now score machine-generated text for everything from content quality assessment to training signal generation. Yet the scoring process is a black box. This paper applies mechanistic interpretability to ask: what actually happens inside a judge model when it emits a score for a summarization task?

Traditional evaluation alternatives are limited: human raters are expensive and unscalable, while ROUGE only measures lexical overlap without understanding semantics. LLM judges can read text, but their decision-making criteria and biases are unknown.

Methods

Evaluation dimensions: readability (fluency, grammar) and adequacy (faithfulness and completeness of information versus the source).
Eight perturbation attacks to create clean vs. corrupted summary pairs: lexical substitution, grammar destruction, information omission, information insertion, redundancy inflation, event-order scrambling, ambiguous coreference, and style drift.
Judge models studied: Themis (based on Llama-3-8B), Prometheus (based on Mistral-7B), plus untuned Llama-3-8B as a baseline.
Interpretability toolkit: logit lens, causal tracing, and attention head knockout.

Key points

Two-stage pipeline: In the first ~15 layers, attention heads perform fast, local comparisons between the candidate and reference summaries and route results to the final sequence position, while MLP contributions are suppressed. Above layer 15, MLPs dominate, integrating signals like a panel of voting judges.
Decision crystallization: The score becomes essentially fixed at a specific depth — layer 26 for Themis and layer 25 for Prometheus. Perturbations before this layer still change the output; after it, the verdict is locked in.
Fine-tuning sculpts, not builds: The untuned Llama-3-8B base model shows a similar two-stage structure, but without clean stage separation (lower-layer MLPs also participate in final decisions) and with later crystallization. Fine-tuning appears to refine an innate capability: suppressing early MLP involvement and moving crystallization earlier.
Specialized attention heads: Knocking out individual heads reveals division of labor — some heads detect readability problems (grammar, fluency), others detect adequacy problems (omissions, factual errors), and others act as information relays.
Critical heads: Heads near the crystallization layer (~25–26) are disproportionately important; removing them collapses judging performance, while removing early-layer heads has milder effects due to redundant pathways.
Security implication: Certain heads appear to function as safety checks; removing them makes the model especially vulnerable to subtle adversarial perturbations such as semantic substitutions — meaning targeted attacks on judge models are conceivable.
Logit-lens 'inner monologue': Early layers produce noisy, unrelated token predictions; middle layers surface quality-related words ("good", "clear", "accurate"); near the crystallization layer, numeric tokens dominate, and the output is effectively settled.

Implications

Evaluation capability has concrete, interpretable internal structure rather than being an inscrutable emergent property — it can be understood, improved, and repaired.
Bias has locatable circuit-level sources (over-sensitive heads, flawed MLP integration, premature crystallization), making it fixable rather than ghostly.
Understanding judge internals enables detecting adversarial manipulation, validating judgment rationale, and building multi-judge systems covering different dimensions.

The post concludes that transparency is a prerequisite for responsible AI evaluation: moving from blind trust to informed trust, since "a judge we don't understand is itself unfair to those being judged."

References cited

Vasava, H., & Jiang, M. (2026). *Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation*. arXiv preprint.
Meng, Y., et al. (2022). *Locating and Editing Factual Associations in GPT*. NeurIPS.
Nanda, N., et al. (2023). Transformer Circuits Thread. Anthropic.
Zou, A., et al. (2023). *Representation Engineering: A Top-Down Approach to AI Transparency*. arXiv preprint.

*Note: the arXiv identifier for the main paper was not provided in the original post.*