Old Ideas, Novel Problems: When LLMs Judge LLM Novelty, Results May Be Unstable
Imagine you've written a paper and want an AI to review whether your idea is novel enough. You open GPT, enter your idea, and it says: "This is a very novel contribution!" You breathe a sigh of relief.
But what if you phrase the question differently—say, telling it "reviewers think this idea is novel," or "reviewers think this idea is not novel"? Would it give the same judgment?
A paper from the Hebrew University of Jerusalem and the Allen Institute for AI says: probably not. And the magnitude of variation is unsettling.
Core Finding: Small Prompt Changes, Big Result Differences
The paper, titled "Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation," builds a systematic controlled experiment testing six LLM judges for stability when evaluating novelty.
The authors mined data from OpenReview—finding submissions where reviewers explicitly affirmed or denied a paper's originality, keeping only extreme cases where reviewer opinions agreed. They then paired these papers with LLM-generated new ideas and had LLM judges score them.
Result: merely changing prompt wording can dramatically alter evaluation outcomes.
For example, "simply telling the judge that reviewers thought an idea was novel"—just that one sentence—can significantly change its judgment. This isn't simply "the model was misled"; the model's judgment itself is highly dependent on contextual framing.
In pairwise evaluations, even a strong model like claude-opus-4-6 ties on 35% of comparisons. claude-sonnet-4-5 is worse—over half of comparisons end in ties. A tie means the model cannot tell which idea is more novel—but with a different prompt, it might immediately distinguish them.
Six Judges, Six Kinds of Instability
The paper tests six judges, including different GPT and Claude versions. Their performance varies widely, but they share one trait: none are stable.
The authors explore multiple dimensions of design choices:
- Pointwise vs. pairwise: scoring individually or comparing in pairs? Both approaches have problems.
- Prompt design: whether to tell the judge reviewers' opinions? Whether to request aggregated judgments?
- Retrieval augmentation: providing relevant papers as reference?
- Reference configuration: what serves as the "ground truth"?
- Aggregation: having the judge make multiple judgments and take a majority vote vs. judging once. Turning off aggregation doubles the tie rate.
- Telling the judge reviewer opinions: a one-sentence difference can flip the direction of judgment.
- Reference set: using human papers vs. LLM-generated ideas as references yields different results.
- Pairwise vs. pointwise: the same idea judged "more novel" in pairwise may score lower in pointwise.
Changes along every dimension affect results. The most surprising finding concerns retrieval—giving judges relevant papers was expected to help them assess novelty better, but "retrieval contributes limited performance gains, and sometimes none at all."
More ironically, in one case a judge without retrieval deemed an idea combining two concepts novel (agreeing with reviewers). But once given papers separately presenting those two concepts, it judged the same idea "a direct combination of existing work"—flipping from "novel" to "not novel."
Retrieval didn't just fail to help—it actively hurt, making the judge more "conservative" and more inclined to judge things as not novel.
The "Judge as the Judged" Dilemma
The paper touches on a deep awkwardness: the LLMs used to evaluate novelty and the LLMs generating ideas are often the same kind of model.
You use GPT to review GPT-generated new ideas. GPT's judgment criteria come from its training data—which papers and ideas it has seen. But GPT also relies on the same training data to generate new ideas. This means the judge's sense of "what counts as novel" and the generator's sense of "what counts as novel" may share the same blind spots.
The paper doesn't directly test this hypothesis, but the observed phenomena fit: judges perform reasonably on human papers (where novelty has reviewer consensus as an anchor), but perform worse on LLM-generated ideas—higher tie rates, lower consistency.
It's like having a student grade another student's answer for "creativity" when both studied from the same syllabus—they may share the same knowledge gaps.
The "Butterfly Effect" in Evaluation Design
The paper's most unsettling finding is the butterfly effect of evaluation design: seemingly trivial choices significantly affect results.
A "Meta" Problem
This paper is itself a "meta" experiment: it uses LLM judges to assess the novelty of LLM-generated ideas, then finds that judging process unstable. But the paper's own pipeline also relies on LLMs—at least in the data construction stage, where LLMs filter and pair ideas.
This creates an interesting recursion: using an unstable tool to study the tool's instability. The authors are candid about this—they don't claim their method is fully objective, instead emphasizing that "instability persists even under careful controls."
This honesty is more persuasive than any claim that "our method is reliable." It tells readers: there is no silver bullet, and any work using LLM novelty evaluation should be read with skepticism.
Implications for Automated Research
As AI-generated papers and AI-reviewed papers make automated research pipelines more common, this paper's findings matter greatly.
If you're building an "automated research" system—having LLMs generate ideas, LLM judges evaluate them, and LLMs select the "most promising" ideas to pursue—you need to know: your filtering stage may be far more fragile than you think. A change in prompt wording can determine which ideas get selected and which get discarded.
The paper's recommendation: at minimum, report all evaluation design choices. Which judge? Pointwise or pairwise? Retrieval? Aggregation? External information given to the judge? This information should be standard configuration, not buried in an appendix.
Honest Uncertainty
The paper ends with a sentence worth remembering: "Until evaluation protocols can account for these sensitivities, automated novelty assessments should be interpreted with significant caution."
This is not a manifesto that "LLMs can't evaluate novelty"—it's more subtle. It says: LLMs can evaluate novelty, but you need to know how fragile your evaluation is. A judge with a 35% tie rate isn't useless—but if you don't know that 35% exists, you'll over-trust its judgments.
It's the same principle as "recall ≠ audit quality": reporting only one dimension (like accuracy) severely overestimates system quality. Novelty evaluation needs multi-dimensional reporting—tie rates, prompt sensitivity, retrieval impact, aggregation effects—so readers can judge the credibility of results.
In an era where AI review is increasingly widespread, this paper reminds us: the reviewer itself needs to be reviewed. And when you review the reviewer, the tools you use may be equally unreliable. This is not a problem that can be solved through infinite recursion—it's one requiring constant vigilance.
---
Paper link: https://arxiv.org/abs/2610.02022
Code repository: https://github.com/simra-shahid/idea_novelty_checker