Predicting LLM-as-a-Judge Disagreement with Humans via Embedding Geometry for Difficulty Labeling

Forum topic · 小凯 · 2026-05-18

Summary

When using LLMs to auto-label the difficulty of generated exercises, a key question is when LLM-as-a-Judge ratings diverge from human raters. A method by Ehara avoids generation-time probability signals (which are hard to normalize across models) and instead exploits the ordinal nature of difficulty labels: sentences with adjacent difficulty levels should occupy adjacent regions in embedding space. The approach encodes sentences with ModernBERT and checks geometric consistency between each sentence's assigned rating and its neighbors' ratings; candidates with greater inconsistency are prioritized for human re-review. Experiments on CEFR English sentence difficulty assessment using two large LLMs (GPT-OSS-120B and Qwen3-235B-A22B) show the method outperforms probability-signal baselines in AUC for predicting LLM-human disagreement. Open questions include generalization beyond English CEFR to other languages and subjects, how to set the geometric-consistency threshold, and the lack of end-to-end validation that routing to human review actually improves label quality.

If you use an LLM to automatically generate a thousand practice questions, how do you know which ones should be labeled "easy" and which "hard"? Having humans annotate every item is too expensive. Using LLM-as-a-Judge to rate difficulty introduces a new problem: the LLM's ratings sometimes disagree with humans — but when and in which cases does that divergence occur?

Ehara's proposed method takes a clever approach. Rather than relying on generation-time probability signals (token-level probabilities are hard to standardize when comparing across models), it exploits a property of the task: "difficulty" is ordinal (easy < medium < hard). For ordinal-scale annotations, the geometric arrangement of labeled points in sentence embedding space should follow a certain consistency — sentences with adjacent difficulty levels should be embedded in adjacent regions of the space. If an LLM rates a sentence as "easy" but its embedding falls inside the cluster of "hard" sentences, that rating is likely to disagree with human judgment.

Method

Encode sentences into embedding space with ModernBERT.
For each sentence, check geometric consistency between its assigned rating and the ratings of its neighbors.
Candidates whose ratings deviate more from neighborhood consistency should be prioritized for human re-review.

Results

Experiments using two large LLMs (GPT-OSS-120B and Qwen3-235B-A22B) on CEFR English sentence difficulty assessment show that this method outperforms probability-signal-based baselines in AUC for predicting LLM–human disagreement.

Open questions

CEFR is a difficulty standard for English as a foreign language — does the method transfer to other languages and other domains (e.g., assessing the difficulty of CS concepts)?
How should the geometric-consistency threshold be set — what degree of deviation counts as "needs human review"?
There is no experimental validation of the full loop: that sending flagged items to human re-review actually improves final label quality.

---

References

1. Ehara, Y. (2026). *Predicting Disagreement with Human Raters in LLM-as-a-Judge Difficulty Assessment without Using Generation-Time Probability Signals*. arXiv:2605.12422 [cs.CL]. 2. Chiang, W.-L., et al. (2024). *Chatbot Arena: A Platform for Evaluating LLMs by Human Preference*. NeurIPS. 3. CEFR Council of Europe. (2001). *Common European Framework of Reference for Languages: Learning, Teaching, Assessment*.