LLMs Aren't Actually Self-Preferential: A Falsification Experiment Strips Away the Self-Bias Label

> "The most dangerous bias is the one we assume exists, and then see everywhere."

---

🎭 Introduction: An Assumed "Fact"

If you follow LLM evaluation, you've likely heard the claim: LLMs exhibit self-preference. When judging text, models favor their own outputs; when revising text, they resist corrections to their own work.

This finding came from several highly-cited 2024 papers and quickly became community consensus. It sounded plausible—humans protect their egos too, and models see so many of their own outputs during training that self-preference seems natural.

But a recent paper by William Guey and Pierrick Bougault offers a surprising answer: under strictly verifiable conditions, LLM self-preference disappears.

Not weakened—disappeared. The gap was -5.1 percentage points, with a 95% confidence interval of [-12.9, +2.7]—statistically indistinguishable from zero.

🔬 What Was Wrong with Previous Research?

Previous self-preference experiments typically worked like this: model A generates text, then model A judges it against text from other models. Result: model A scores its own text higher.

The problem—this design conflates two variables:

1. Authorship: whether the model wrote the text itself 2. Text quality: the model's own text may genuinely have certain stylistic features

If a model isn't protecting its ego but simply recognizes and prefers familiar styles, the score difference reflects style familiarity, not authorship bias.

A deeper issue: who defines what counts as a "good revision"? Previously, another LLM did—but model-judging-model is itself noisy.

🧪 The New Experiment: IFEval as Referee

The design's cleverness: a deterministic verifier replaces LLM judges.

The procedure:

1. Draft generation: the model is given an instruction (e.g., "write a summary of no more than 50 words") and produces a draft 2. Violation verification: IFEval's official checker confirms the draft violates a constraint 3. Verified correction: a correction is prepared that IFEval confirms fixes the violation 4. Author vs. novice decision: the model decides whether to accept the correction, with two key conditions:

• Author condition: the model acts as the draft's "original author," seeing its own draft in context
• Novice condition: a fresh model (that didn't write it) sees the same draft

If models are self-preferential, acceptance rates should be lower in the author condition.

Who defines "valid"? IFEval's deterministic checker—code, not another model. This eliminates the circular noise of model-judging-model.

📊 Four Models, 85 Comparisons, Zero Evidence

Experiment scale:

• 4 mid-tier model families (covering different architectures and training paradigms)
• 85 author-vs-novice comparisons
• 3 progressive experimental rounds (pilot to formal)

Core result:

> Author models rejected verified-correct corrections at essentially the same rate as novice models. Gap: -5.1 percentage points, 95% CI [-12.9, +2.7].

The interval crosses zero, meaning "self-preference exists" cannot be statistically distinguished from "no self-preference." Even if a real effect exists, it's no larger than ~13 percentage points.

The result was consistent across all four models—a stable cross-model pattern, not one model's fluke.

🧠 The Most Interesting Finding: The "Character" of Rejections

Though self-preference disappeared quantitatively, the character of rejections differed qualitatively between author and novice models.

When author models rejected a verified-correct correction, 97% of rejection reasons were "flaw-catching," not "preference".

In other words, author models weren't saying "I just like my version"—they were saying "your correction introduces new problems." They were nitpicking, not defending their ego.

This distinction matters: flaw-catching is about the revision itself; preference is about authorship. Flaw-catching may be useful—an author model, having written the text, may genuinely be more sensitive to problems introduced by corrections.

But this finding warrants caution. The paper explicitly labels it a "qualitative observation," not a quantitative conclusion. It cannot rule out the alternative explanation that author models' "flaw-catching" is merely a rationalized excuse.

🤔 Why This Matters

1. Falsification Is Scarcer Than Confirmation

In AI research, "discovering a new bias" is easier to publish than "falsifying an old one." But scientific progress depends on falsification—a field that cannot be falsified is not science. Guey's team did something rare: testing an accepted "fact" with a more rigorous design.

2. A Paradigm for Experimental Design

The methodology offers lessons for all LLM bias research:

• Deterministic verifiers instead of LLM judges: eliminates judge bias
• Separating authorship from text quality: novice models seeing the same text controls for quality
• Three progressive rounds: gradual scaling avoids one-shot flukes
• Pre-registered effect sizes: explicitly stating "effects below 13 pp cannot be excluded," rather than post-hoc p<0.05 thresholds

This mirrors the spirit of randomized controlled trials in medicine—not "finding an effect," but "bounding the effect."

3. A Warning for LLM-as-Judge

If self-preference vanishes under verifiable conditions, what was previously observed? A likely explanation: models don't favor their own work—they favor familiar styles. When judging free-form text, style familiarity correlates highly with authorship.

Practical implication for LLM-as-Judge: if you're worried about judge bias, attending to style diversity may matter more than authorship.

4. Micro-Practice of Scientific Integrity

A striking detail: in small-scale pilots, the authors observed a "self-skepticism hint"—author models seemed more suspicious of their own drafts. The effect failed to replicate at scale, and they openly reported "failed to replicate" rather than packaging it as a new finding.

🧭 Honest Boundaries

The paper's limitations are clearly stated:

• Only mid-tier models tested: whether frontier models (GPT-4, Claude 3.5-class) show self-preference remains untested
• Only IFEval-style constraints: self-preference on stylistic or content revisions remains untested
• Sample size limits: 85 comparisons exclude effects >13 pp, but smaller effects may exist
• The "flaw-catching" finding is qualitative: the 97% figure comes from qualitative analysis

🌌 A Feynman-Style Closing

Feynman famously said: "The first principle is that you must not fool yourself—and you are the easiest person to fool."

LLM self-preference was a story we told ourselves. It was too plausible—so smooth that we forgot to test it.

Guey's team didn't discover a new fact; they took an assumed fact, subjected it to stricter scrutiny, and watched it disappear. This work's value lies not in what it "discovered," but in what it "ruled out."

Science is not about what we prove—it's about what we exclude. LLMs aren't self-preferential. That negative sentence may be closer to the truth than any positive one.

---

Paper: https://arxiv.org/abs/2606.20093

Code: https://github.com/williamguey/self-preference-revision

Authors: William Guey, Pierrick Bougault (Tsinghua University)