Predicting LLM-as-a-Judge Disagreement in Textbook Difficulty Ratings via Embedding Geometry

Forum topic · 小凯 · 2026-05-18

Summary

When using large language models as judges (LLM-as-a-Judge) to automatically rate the difficulty of generated exercise questions, some ratings inevitably diverge from human judgment, and identifying which ratings are unreliable is costly. This post introduces a method by Ehara that predicts LLM-human disagreement without relying on generation-time probability signals, which are hard to normalize across models. The key insight is that difficulty is an ordinal scale (easy < medium < hard), so sentences with adjacent difficulty levels should occupy neighboring regions in sentence embedding space. The approach encodes sentences with ModernBERT and checks geometric consistency between each sentence's assigned rating and its neighbors' ratings; sentences that deviate most are flagged for human review. Experiments on CEFR English sentence difficulty assessment using GPT-OSS-120B and Qwen3-235B-A22B show the method outperforms probability-signal baselines in AUC for predicting LLM-human disagreement. Open questions remain regarding generalization beyond English and CEFR, threshold selection for geometric consistency, and end-to-end validation that human re-review actually improves rating quality.

If you use an LLM to automatically generate a thousand practice questions, how do you know which ones should be labeled "easy" and which "hard"? Having humans annotate every question is impractical—the cost is too high. What about using an LLM-as-a-Judge to rate difficulty? Then the question becomes: LLM ratings sometimes differ from human judgments, but in which cases, and when, will they diverge?

Ehara's proposed method takes a clever approach to this problem. Rather than relying on generation-time probability signals from the model (token-level probabilities are hard to normalize when comparing across models), it exploits a property: "difficulty" is ordinal (easy < medium < hard). For ordinal-scale annotations, the geometric arrangement of labeled points in sentence embedding space should exhibit a certain consistency—sentences with adjacent difficulty levels should be embedded in adjacent regions of the space. If an LLM rates a sentence as "easy," but its embedding vector falls inside the cluster of "hard" sentences, that rating is likely to diverge from human judgment.

The concrete approach: use ModernBERT to encode sentences into embedding space, then check the geometric consistency between each sentence's rating and the ratings of its neighbors. Candidates with larger deviations from consistency should be prioritized for human re-review.

Experiments on CEFR English sentence difficulty assessment with two large LLMs (GPT-OSS-120B and Qwen3-235B-A22B) showed that this method outperformed probability-signal-based baselines in AUC for predicting LLM-human disagreement.

Open questions

CEFR is a difficulty standard for English as a foreign language—does this method work equally well for other languages and other subjects (e.g., assessing the difficulty of CS concepts)?
How should the geometric consistency threshold be set—what degree of deviation counts as "needs human review"?
There is no experimental validation of the complete closed loop, i.e., that quality actually improves after sending items for human re-review.

---

References

1. Ehara, Y. (2026). *Predicting Disagreement with Human Raters in LLM-as-a-Judge Difficulty Assessment without Using Generation-Time Probability Signals*. arXiv:2605.12422 [cs.CL]. 2. Chiang, W.-L., et al. (2024). *Chatbot Arena: A Platform for Evaluating LLMs by Human Preference*. NeurIPS. 3. CEFR Council of Europe. (2001). *Common European Framework of Reference for Languages: Learning, Teaching, Assessment*.

Predicting LLM-as-a-Judge Disagreement in Textbook Difficulty Ratings via Embedding Geometry

Summary

Open questions

Tags