Rubrics Over Intelligence: Small LLM Judges Match GPT-5 Under Detailed Rubric Anchoring

Background

Why do two teachers grading the same student essay often give different scores? Educational research has established a clear answer over decades: inter-rater agreement depends on how detailed the rubric is. Vague criteria like "is the writing good?" produce wide disagreement, while specific rubrics such as "is the thesis clear (0-3)? Are the supporting arguments sufficient (0-3)? Is the language fluent (0-3)?" drive much higher consistency.

A new paper argues that the same principle applies to large language models acting as judges, and the implication is counterintuitive: a small model used as judge, paired with a sufficiently detailed rubric, performs as well as GPT-5.

The paper introduces the any-to-bench framework. Its core design principle: a frontier model is used only once to extract items and rubrics into a benchmark; all subsequent scoring is delegated to low-cost models.

Experimental Design

The experimental setup is rigorous:

• 6 judge models drawn from two model families, each tested at three reasoning effort levels (low / medium / high)
• 2 respondent models (a "small" tier and a "medium" tier), each at three reasoning effort levels
• 24 open-ended items spanning proofs, essays, translations, and drawing tasks
• 3 scoring passes per item per judge model, yielding 3,456 scoring records

Agreement is measured with ICC(2,1), which quantifies how much of the score variance comes from the answers themselves versus the raters. ICC = 1 means judges are fully interchangeable; ICC = 0 means scores depend entirely on who is judging.

Key Findings

Finding 1: Answers explain 95.6% of variance; raters explain only 0.2%

This is the most striking result. Under rubric anchoring, answer identity explains 95.6% of score variance while judge identity explains only 0.2%.

The practical implication: regardless of which model serves as judge, as long as a rubric is available, the score is almost entirely determined by answer quality, not by rater identity.

As a comparison, increasing a respondent's reasoning effort can raise scores by up to 0.143 (fraction of full marks), but increasing a judge's reasoning effort shifts scores by at most 0.006. The marginal value of a respondent thinking harder is 24 times that of a judge thinking harder.

Finding 2: The reference answer is what actually matters in a rubric

Ablations progressively strip components from the rubric:

1. Full rubric (scoring criteria + answer-level descriptors + official reference answer): ICC = 0.888 2. Remove scoring criteria and level descriptors, keep only the reference answer: ICC unchanged — no measurable difference 3. Remove the reference answer too: ICC collapses from 0.888 to 0.628, scores inflate, and judge reasoning effort becomes important again

The conclusion is clear: the part of the rubric that actually matters is the official reference answer, not the 0-3 descriptor definitions. With a reference answer, small models can score reliably; without one, even large models fall back on "vibes."

This mirrors human grading: give teachers a reference answer and they produce stable scores; give them only "is the writing good?" and inter-rater disagreement spikes.

Finding 3: One judge is enough

Moving from 2 to 6 judges produces almost no change in mean panel reliability. Multiple passes do not add measurable information.

Under rubric anchoring, a single judge's judgment is already sufficiently reliable. Additional judges are merely insurance against an unlucky draw, not a genuine aggregation requirement.

Two Biases That Did Not Emerge

The paper also checks two classic LLM-judge biases:

Length bias. LLM judges often favor longer answers. Under rubric anchoring, no length premium appears. Within a single respondent, the length-score correlation is +0.6 to +0.7, but across respondents the advantage disappears — short, well-written answers score just as well as long ones.

Family bias. LLM judges often favor answers from their own model family. Under rubric anchoring, no family bias emerges.

These null results matter. Prior literature extensively documents LLM-judge biases (length, order, self-preference), but those studies typically use pairwise preference settings: present two answers and ask which is better. This paper uses absolute scoring: present one answer plus a rubric and ask for a score. The bias structure of the two settings may be fundamentally different.

Why This Matters

First, it challenges the prevailing LLM-as-Judge practice. Current benchmark evaluation routinely calls GPT-4 or Claude as judge. This paper argues: with a good rubric, a far cheaper small model suffices. Evaluation costs drop directly.

Second, it redefines what determines evaluation quality. The community has focused on "which model is the judge." This paper relocates the lever to "how well the rubric is designed" — a labor-intensive, one-time investment rather than a compute-intensive, recurring expense.

Third, the 95.6% variance result carries a deeper implication. Under well-designed evaluation, judge "intelligence" is an over-attended dimension. What matters is respondent capability and evaluation framework design; the judge is merely an executor. This aligns with the principle that specialization beats generalization: use the right tool for the right task rather than hoping a single all-powerful judge can do it all.

Honest Limitations

Dataset scope. The experiments use examination items from Taiwan, with 24 items across diverse types. The scale is limited; generalization to larger and more diverse benchmarks requires further verification.

Reference answer availability. Not every open-ended problem has a reference answer. For genuinely open tasks such as creative writing or open discussion, the framework's applicability is uncertain.

ICC absolute value. While 0.888 is respectable, it may still be insufficient for high-stakes contexts such as university admissions. The paper also notes that a panel of six frontier-model judges is no more reliable than a small model — reinforcing that the bottleneck is the evaluation framework, not the judge.

Conditional validity of the "one judge is enough" conclusion. This holds under rubric anchoring. Without a good rubric, multi-judge voting remains an effective variance-reduction strategy.

Closing Perspective

The paper surfaces a more general principle: a tool's required intelligence should match the task's required intelligence.

Using GPT-5 to "compare an answer against a reference and assign a score" is like using a microscope to hammer a nail — a fine tool, but the task does not demand it. What a rubric does, at its core, is decompose a high-cognitive-load judgment task into a series of low-cognitive-load matching tasks. Once the task is reduced to matching, small models suffice.

This echoes findings from several recent papers on task decomposition: delegating reasoning to deterministic tools, using small models for first-pass filtering with large models for review, and splitting context engineering into research-plan-implement stages. The common thread is specialization — reserving high intelligence for the parts that genuinely require it and offloading the rest to smaller models or deterministic systems.

The paper's title is itself the best summary: scoring needs a rubric, not intelligence. The lesson generalizes: good workflow design beats good executors. Whenever a workflow step seems to demand top-tier intelligence to function, the first thing to check is whether the workflow itself is well designed.

---

Paper: https://arxiv.org/abs/2608.17938

HTML full text: https://arxiv.org/html/2608.17938v1

Open-source code: https://github.com/JacobLinCool/any-to-bench