Rubrics Over Intelligence: Small Models Judge as Well as GPT-5

Forum topic · ✨步子哥 · 2026-08-19

Summary

This zhichai.net forum post introduces any-to-bench, a benchmark framework from a Taiwan-based research team (arXiv:2608.17938) showing that LLM-as-judge scoring quality depends on rubrics, not model intelligence. Across 24 open-ended questions, 6 judge models, 2 answer models, and 3,456 scoring records measured with ICC(2,1), rubric-anchored scoring explained 95.6% of score variance by answer identity versus only 0.2% by judge identity, achieving ICC of 0.888. Ablations reveal the reference answer—not the score-level definitions—is the active rubric component: removing it drops ICC to 0.628. Small models with detailed rubrics match GPT-5 as judges, one judge suffices, and no length or family bias appears under absolute scoring. The post discusses cost implications, limitations, and parallels with task-decomposition research.

Have you ever wondered why different teachers can give very different scores to the same student essay?

Education research has studied this for decades, with a clear conclusion: scoring consistency depends on the detail level of the rubric. Vague criteria like "is it well written?" produce large inter-rater disagreement; detailed rubrics—"is the thesis clear (0–3 points)?", "is the evidence sufficient (0–3 points)?"—shrink disagreement dramatically.

A new paper applies this insight to LLM-as-judge evaluation, with a counterintuitive implication: a small model judge, given a sufficiently detailed rubric, performs as well as GPT-5.

The paper comes from a Taiwan-based research team and proposes the any-to-bench framework, built on a simple principle: use frontier models once, at benchmark-construction time (extracting questions and rubrics), then hand all judging to low-cost models.

Experimental design

6 judge models: two model families × three reasoning-effort levels
2 answer models: GPT-5.6 Luna and Claude Sonnet 5, each at three reasoning-effort levels
24 open-ended questions: proofs, essays, translation, drawing, etc.
3 judgments per question per judge model, totaling 3,456 scoring records

Reliability is measured with ICC(2,1) (intra-class correlation), which captures how much score variance comes from the answers themselves rather than the judges. ICC = 1 means judges are fully interchangeable; ICC = 0 means scores depend entirely on who is judging.

Three core findings

Finding 1: The answer explains 95.6% of the score; the judge explains 0.2%

This is the paper's most striking number. Under rubric anchoring, answer identity explains 95.6% of score variance; judge identity explains only 0.2%.

In other words: whichever model judges, as long as it has a rubric, the score is almost entirely determined by how good the answer is, not by who scores it.

For comparison: raising the answer model's reasoning effort improves scores by up to 0.143 (proportion of full score), while raising the judge's reasoning effort changes scores by at most 0.006. The value of the answer model thinking harder is 24× that of the judge thinking harder.

Finding 2: What matters in a rubric is the reference answer

Ablating rubric components step by step:

1. Full rubric (scoring criteria + score levels + official answer): ICC = 0.888 2. Removing criteria and levels, keeping only the official answer: ICC unchanged—no measurable difference 3. Removing the official answer too: ICC collapses from 0.888 to 0.628, scores inflate, and judge reasoning effort becomes important again

The conclusion is clear: the working component of the rubric is not the "0–3" level definitions but the official reference answer. With a reference answer, small models score by comparison; without one, even large models fall back on vibes.

Finding 3: One judge is enough

Going from 2 judges to 6 judges barely changes average panel reliability, and repeated scoring adds no measurable information. Under rubric anchoring, one judge's judgment is already reliable. Extra judges are just insurance against a bad draw, not a source of new information.

Two biases that did not appear

Length bias: LLM judges typically favor longer answers. Under rubric anchoring, there is no length premium in cross-answer comparisons—within an answer model, length-score correlation is +0.6 to +0.7 (longer answers genuinely score higher), but across answer models the length advantage disappears. Well-written short answers score just as high.

Family bias: LLM judges typically favor answers from their own model family. Under rubric anchoring, no family bias is detected.

These null results matter because prior literature documented length, position, and self-preference biases—but mostly under pairwise preference settings. This paper uses absolute scoring with a rubric, and the bias structure may be fundamentally different in that setting.

Why this matters

1. It challenges the default of using frontier models as judges. If you have a good rubric, much cheaper small models suffice—directly cutting evaluation costs. 2. It relocates the lever of evaluation quality. The key question is not "which model judges" but "how well is the rubric designed." Rubric design is labor-intensive but one-off; judge intelligence is compute-intensive and paid every run. 3. "Answers explain 95.6% of variance" has a deeper meaning. Judge intelligence is an overrated dimension. What matters is the answer model's ability and the evaluation framework's design—the judge is just an executor.

Honest limitations

Dataset scale: 24 exam questions across several types is limited; generalization to larger, broader evaluations needs verification.
Availability of reference answers: tasks like creative writing lack standard answers; the framework's applicability there is questionable.
Absolute ICC value: 0.888 is good but may be insufficient for high-stakes uses such as college admissions.
"One judge suffices" is conditional: it holds only under rubric anchoring. Without good rubrics, multi-judge voting remains a valid variance-reduction tool.

My take

The paper illustrates a general principle: a tool's required intelligence should match the task's required intelligence. Using GPT-5 to score against a reference answer is like using a microscope to hammer nails. A rubric decomposes the high-cognition task of "judging" into a series of low-cognition "matching" tasks—and once a task is at the matching level, small models suffice.

This echoes other findings on division of labor: outsourcing reasoning to Prolog (Euclid-MCP), small-model triage with large-model review (Rebucca), and staged context engineering (ACE). It also illustrates the "evaluation blind-spot law": when a dimension (judge intelligence) barely affects results, either the rubric has already absorbed that judgment space, or you haven't yet tested a scenario where it matters—the paper honestly flags the latter.

The paper's title is its best summary: scoring needs a rubric, not intelligence. When a step seems to require top-tier intelligence to do well, first check whether the process design is the problem.

Paper: https://arxiv.org/abs/2608.17938

HTML full text: https://arxiv.org/html/2608.17938v1

Open-source code: https://github.com/JacobLinCool/any-to-bench