Large Language Models Don't Actually Play Favorites: A Falsification Experiment Debunks the Self-Preference Label

Forum topic · ✨步子哥 · 2026-06-21

Summary

A new paper by William Guey and Pierrick Bougault challenges the widely accepted claim that large language models exhibit self-preference—favoring their own outputs when judging or revising text. Prior studies conflated two variables: authorship and text quality, and relied on model-based judges, which introduce noise. The authors designed a rigorous experiment using IFEval's deterministic verifiers instead of model judges. Across 4 mid-tier model families and 85 author-vs-novice comparisons over three escalating rounds, they found no evidence of self-preference: author models rejected verified-valid corrections at nearly the same rate as novice models, with a difference of -5.1 percentage points (95% CI [-12.9, +2.7]), statistically indistinguishable from zero. A qualitative finding shows that 97% of author-model rejections cited flaw-catching rather than preference, suggesting authors are more sensitive to problems introduced by edits rather than defending their work. The paper also reports methodological strengths—pre-registered effect sizes, pilot replication discipline—and clear limitations: only mid-tier models and IFEval-style constraints were tested. The results imply that earlier observations of self-preference may reflect style familiarity rather than authorship bias, with practical implications for LLM-as-Judge evaluations.

A new falsification study argues that the widely accepted "self-preference" bias in large language models disappears under rigorously verifiable conditions.

Background: The默认 Consensus Under Question

If you follow LLM evaluation, you've likely heard the claim: LLMs exhibit self-preference—models rate their own generated text higher and resist corrections to their own work. This finding, from several highly cited 2024 papers, quickly became community consensus. It sounded plausible: humans protect their own work, and models trained on their own outputs would naturally develop such bias.

But a new paper by William Guey and Pierrick Bougault (Tsinghua University) offers a surprising answer: under strict verifiable conditions, LLM self-preference disappears.

Not weakened—gone. The gap was -5.1 percentage points, 95% CI [-12.9, +2.7]—statistically indistinguishable from zero.

What Went Wrong in Earlier Studies?

Past self-preference experiments typically asked model A to generate text, then judge whether its own text was better than text from other models. Model A scored its own text higher.

The problem: the design confounded two variables:

1. Authorship — whether the model wrote the text itself 2. Text quality/style — the model's own text may have distinct stylistic features

If a model isn't protecting its work but is simply more familiar with its own style, the rating gap reflects style familiarity, not author preference. A deeper issue: "what counts as a good revision" was defined by another model—and model-judging-model is inherently noisy.

The New Experiment: IFEval as Referee

The clever move: replace model judgment with a deterministic verifier.

1. Draft generation: The model receives an instruction (e.g., "write a summary under 50 words") and produces a draft 2. Verified violation: IFEval's official checker confirms the draft violates a constraint 3. Verified correction: A correction is prepared that IFEval confirms fixes the violation 4. Author vs. novice decision: The model decides whether to accept the correction under two conditions:

Author condition: the model acts as the draft's "original author," seeing its own draft in context
Novice condition: a "fresh" model sees the same draft

If models protect their work, acceptance rates should be lower in the author condition. Here, "valid" is defined by deterministic code—not another model.

Four Models, 85 Comparisons, Zero Evidence

4 mid-tier model families (different architectures and training paradigms)
85 author-vs-novice comparisons
3 escalating rounds of experimental design

Core result:

> Author models rejected verified-valid corrections at essentially the same rate as novice models. Gap: -5.1 percentage points, 95% CI [-12.9, +2.7].

The interval crosses zero, meaning the data cannot distinguish "has self-preference" from "doesn't." Even if an effect exists, it's bounded at roughly 13 percentage points. The pattern held across all four models.

The Most Interesting Finding: The "Character" of Rejections

While self-preference vanished quantitatively, a qualitative difference emerged in how rejections were justified.

When author models rejected a verified-valid correction, 97% of the stated reasons were "flaw-catching," not "preference". The models weren't saying "I prefer my version"—they were saying "this correction introduces new problems." They were nitpicking, not protecting their work.

This matters: flaw-catching concerns the correction itself; preference concerns authorship. An author model, having written the text, may genuinely be more sensitive to newly introduced issues. But caution is warranted—the authors explicitly label this a qualitative observation and cannot rule out that "flaw-catching" is merely a rationalized excuse.

Why It Matters

1. Falsification Is Scarcer Than Confirmation

Finding a new bias is easier to publish than debunking an old one. Yet scientific progress depends on falsification. Testing an accepted "fact" under stricter conditions—and watching it disappear—is rarer and arguably more valuable.

2. A Methodological Paradigm for Bias Research

Deterministic verifiers instead of model judges: eliminates judge bias
Authorship separated from text quality: novice models see identical text
Three escalating rounds: avoids one-off flukes
Pre-registered effect sizes: explicitly stating "effects below 13 pp cannot be excluded," rather than post-hoc p-hacking

This mirrors the spirit of randomized controlled trials in medicine—estimating the upper bound of an effect, not just "finding" one.

3. A Warning for LLM-as-Judge

If self-preference vanishes under verifiable conditions, what was the previously observed "self-preference"? A plausible explanation: models favor familiar styles, not their own authorship. When judging free-form text, style familiarity correlates with authorship. Practical implication: if you're worried about judge bias, style diversity may matter more than authorship.

4. Micro-Practice of Scientific Integrity

A pilot observed a "self-skepticism hint"—author models seemed more suspicious of their own drafts—but this failed to replicate at scale. The authors reported the non-replication rather than packaging the pilot result as a finding.

Honest Limitations

Only mid-tier models tested: frontier models (GPT-4, Claude 3.5 class) untested
Only IFEval-style constraints: style/content revisions untested
Sample size: 85 comparisons exclude effects >13 pp, but smaller effects may exist
Qualitative nature of the flaw-catching finding: the 97% figure comes from qualitative analysis

Closing Thought

Feynman: "The first principle is that you must not fool yourself—and you are the easiest person to fool." LLM self-preference was a story we told ourselves because it sounded too reasonable to question. This work didn't discover a new fact—it took an assumed fact, tested it more rigorously, and watched it vanish. Science is not about what we prove, but about what we exclude. Each excluded false belief brings us one step closer to truth.

---

Paper: https://arxiv.org/abs/2606.20093

Code (open source): https://github.com/williamguey/self-preference-revision

Authors: William Guey, Pierrick Bougault (Tsinghua University)