A Familiar Experience
Ask GPT to write sadness and it gives you:
> She stood by the window, filled with endless sorrow. Tears slid silently down her face as she felt a piercing loneliness.
You frown. Not because it's badly written—grammar is fine, the emotion is present. Something just feels off. A human writer might give you:
> She put the kettle on the stove and watched the flame lick the bottom. The water boiled. She didn't move. It boiled over, extinguishing the flame, and the hissing went on in the kitchen for a long time. She still didn't move.
The difference? The first passage writes the word "sadness" directly onto the page. The second never uses the word, but leaves you chilled.
Literary writing has an old maxim: "Show, don't tell"—canonized from Hemingway to Chekhov. But have you considered: when an LLM serves as a judge, a reward model, or an automated editor, can it recognize "showing"?
In September 2026, independent researcher Levent Bulut published a paper (arXiv:2609.20712) offering an answer precise enough to be unsettling: LLMs don't merely fail to recognize "showing"—they have a systematic directional preference, always toward "telling." This preference has a name: Summarization Bias.
Not Just "Bad Writing"
First, a clarification: this paper is not arguing "LLMs write poorly." That would be boring—models obviously can't, just as models once couldn't play Go.
The core claim is directionality.
Imagine an axis. One end is "told mode": emotion and information written directly on the text's surface, requiring no inference from the reader. "She was sad" is told. The other end is "shown mode": the same content pressed beneath the surface, encoded in physical detail, action, environment—the reader must reconstruct it. "The kettle boiled over and she didn't move" is shown.
If an LLM were merely a "weak model," its errors along this axis should be symmetric—sometimes over-telling, sometimes over-obscuring, no net drift.
But Bulut's claim is: the errors are asymmetric. They always point toward telling.
This asymmetry is the paper's real thesis. Symmetric weakness is "capability shortfall." Asymmetric weakness is "systematic bias." Capability problems can be solved with scale; systematic biases don't disappear as models grow—they may be amplified.
Two Regimes: Generation and Judging
Summarization Bias operates in two regimes.
The generation regime: you ask a model to write with "showing," and it slides back into "telling." You instruct: "Don't use the word 'sad'—let the reader infer it." It nods, then delivers "her eyes flickered with silent sorrow"—still telling, just with a synonym.
The judging regime: you ask a model to score two passages—one told, one shown—with identical content and length. It systematically scores the told version higher.
The paper's key judgment: the judging regime is more dangerous.
Why? Generation bias is only a quality ceiling—it affects only the machine's own prose. Judging bias is a selection pressure.
Consider today's reality: in RLHF training, the reward model is an LLM judge. Automated scoring, AI editing tools, content moderation—all LLM judges. If an LLM judge systematically prefers told mode, what happens?
Text optimized to please that judge drifts toward telling, generation after generation. Each is flatter, more explicit, with everything on the surface than the last. This isn't random noise—it's a directional gradient pulling narrative from "showing" toward "telling."
Bulut calls this "drift toward told mode." If it holds, it means: LLM judges are applying a quantifiable downward pressure on human writing, pushing text toward the opposite of what they're supposed to evaluate.
The Failure of Five Annotators
The paper's weightiest evidence comes from a completed three-study reliability report (arXiv:2609.13936).
Design: a rule-based detector and four LLMs annotated features in "shown mode" text, compared against blinded human annotators. The corpus was Turkish narrative text.
Results split into two tiers by feature type:
Surface features (physical markers, temporal markers, etc.): machines caught them all. But these features are nearly universal in the corpus, so catching them has no diagnostic value—like a weather forecaster saying "there will be air tomorrow": 100% accurate, zero information.
Inference features (requiring annotators to recognize "an abstract emotion materialized into a concrete object"—the diagnostic signature of shown mode): all five machine annotators failed.
The critical feature is "materialized metaphor." Across 100 scenes, human annotators counted 9. The five machine counts: 0, 1, 40, 72, 78. Cohen's κ for four of the five was indistinguishable from random guessing.
Note this distribution. Not "all machines undercounted" or "all overcounted"—they were mutually inconsistent too. One said 0, another said 78. That inconsistency is itself the clue: the machines weren't "recognizing" the feature—they were guessing, and guessing in different directions.
Bulut's interpretation is restrained: the result is "consistent with" summarization bias, but "consistent with" is not "confirms." He explicitly notes it could also mean the rule definitions aren't operationalized enough for any annotator—including humans—to apply consistently.
But one fact is certain: machines work at the surface layer and fail entirely at the inference layer. They aren't "unreliable in places"—they fail at the shown-mode layer, precisely the layer where Summarization Bias predicts collapse.
This isn't "the model isn't strong enough." This is "the model is systematically blind in a specific direction."
Pre-registration: Sealing Off the Exits
The paper's most unusual feature isn't its claim but its methodological stance.
Most conceptual framework papers propose a construct, then offer examples saying "see, plausible." Bulut did something different: he pre-registered a validation protocol and fixed, in advance, the conditions under which the construct would be abandoned.
The protocol has four stages:
Stage 1: Construct 20 matched scene pairs. Each pair has a told version and a shown version, identical content, length-matched (±5%), same Flesch-Kincaid reading level. Two independent annotators code each version's SI (Suppressed Information Index). Categories with Cohen's κ < 0.60 are dropped immediately—no redefinition until they pass.
Stage 2 (generation test): Give ≥3 model families an explicit "shown mode" instruction and have them generate text. Compute each output's SI, compare against the human shown-version target and told-version baseline. Prediction: model SI significantly below the human shown target, near the told baseline. If model SI is not significantly lower, generative Summarization Bias fails.
Stage 3 (evaluation test, the core): Give each told/shown pair to the same models as judges for scoring. Collect human judges' judgments too. Prediction: the rate at which models prefer the told version exceeds the human rate, with effect size Cohen's h ≥ 0.3 or odds ratio ≥ 2. If model and human preference rates don't differ, or models prefer shown mode, Summarization Bias fails and the construct is withdrawn.
Stage 4 (differential checks): Deliberately length-mismatch versions—if preference vanishes, it's verbosity bias, not Summarization Bias. Change prompt framing—if preference vanishes, it's sycophancy. Only passing both checks confirms Summarization Bias.
The key sentence after Stage 4: "If Stage 3 falsifies H1e, the construct is withdrawn, and that withdrawal is published with the same prominence as a confirmation would have received."
This is nearly unseen in AI research. Most papers' "limitations" sections are decorative—listing "future work" and moving on. This paper's limitations are operational—they define the construct's life-or-death conditions, fixed before data collection.
Why Directionality Matters More Than Strength
One passage deserves quoting:
> "Symmetry versus asymmetry is not a matter of degree but of kind, and it is the difference that makes the construct testable."
A symmetrically weak model can be fixed with scale—more parameters, more data, more training, and symmetric errors shrink. An asymmetrically weak model won't be fixed by scale, because the bias is structural, not capability-based.
This is the same lineage as the "evaluation blind spot" law: toxicity classifiers produce false positives on harm laundering because their blind spot is structural—they learned to recognize "surface toxicity vocabulary," not "semantic-level asymmetric discrimination." Bigger models get better at surface vocabulary without gaining the deeper ability.
Summarization Bias predicts the same structural blind spot: models learned "surface narrative vocabulary" (emotion words, metaphors, adjectives) but not "reconstruction of inferential structure." As judges, they reward "surface vocabulary present" and penalize "reconstruction required"—even at identical word counts.
Scale won't fix a structural blind spot. Scale will just run you faster in the wrong direction.
The Judge's Selection Pressure: A Slow-Motion Disaster
Let me expand on the judging regime's risk, the paper's most practical and unsettling part.
Imagine a content platform using LLM judges to score user content, with high scores earning more recommendations. Writers quickly discover: articles that state emotions explicitly score higher than those that show them implicitly. So they adjust—more adjectives, more emotion-naming, less restraint.
Generation one: slight difference. Generation two: obvious. Generation ten: the platform is all "she felt a deep sadness, tears streaming." Shown mode gets squeezed out by selection pressure.
This isn't hypothetical. RLHF reward models are LLM judges. AI editing tools are LLM judges. Automated essay scoring, content moderation, A/B copy selection—all LLM judges. Each exerts selection pressure on evaluated text. If that judge has Summarization Bias, the pressure points from showing toward telling.
A biased judge is not neutral—it is a directional evolutionary pressure.
This is isomorphic to "harm laundering": safety training didn't eliminate discrimination, just changed its form—the toxicity classifier says "safe" while REGARD says "worse." An LLM judge doesn't make writing better, just flatter—the judge says "score up" while the SI index says "inferential structure gone." The proxy objective (score) improves; the true objective (narrative quality) degrades.
The Paper's Own Limitations
Fairness requires noting the limitations, which the author explicitly lists:
- No data yet. This is a conceptual framework plus a pre-registered protocol, not a validated result.
- Supporting evidence is suggestive, not confirmatory. The completed reliability study is "consistent with" Summarization Bias but "equally consistent with" overly loose rule definitions.
- Definition dependence. Told-shown coding and SI both depend on annotator judgment; without Stage 1's κ checks, downstream tests are uninterpretable.
- Limited scope. Any confirmation would be bounded by model family, language, and genre—not generalizable to "all LLMs."
- Construct proximity. Summarization Bias is designed to be separable from verbosity bias and sycophancy, but that separability is itself a Stage 4 hypothesis, not a precondition.
My Take: Why This Paper Deserves Serious Attention
This paper belongs to a rare genre in AI research: it proposes a falsifiable construct and fixes the falsification conditions before data collection.
Most AI evaluation papers are retrospective—run the experiments first, extract a story from the results afterward. The problem isn't dishonesty but that retrospective stories can't be falsified—if a different dataset or model gives different results, you can always find a new story.
Bulut did the opposite: define the construct, fix the test conditions, fix the abandonment conditions, then start collecting data. This is clinical medicine's pre-registered trial methodology, nearly unseen in AI research. The methodological stance alone deserves respect—regardless of how Stage 3 turns out.
What interests me more is the construct's scope. Summarization Bias is defined in a literary context, but its core—models systematically prefer "surface statements" over "inferential structure," and this preference becomes selection pressure when they judge—may extend far beyond literature.
Think code review. An LLM judge might prefer "well-commented, self-explanatory variable names" over "elegantly structured but requiring two reads." The former is told mode; the latter is shown mode.
Think academic writing. An LLM judge might prefer "papers with explicit thesis statements opening every section" over "theses implicit in the argument's structure."
Think product copy. An LLM judge might prefer "bullet-pointed selling points" over "scenarios letting users infer the benefits."
Any domain with a "surface statement vs. inference reconstruction" tension may harbor Summarization Bias. And that tension exists in nearly all human-created content.
If Stage 3 validates the construct, the question isn't "LLM judges aren't good enough" but "LLM judges are directionally shaping human expression." The direction: toward flatter, more explicit, less reader-participatory text. Not toward "better"—toward "easier for LLM judges to detect."
This is another instance of the proxy objective trap: we optimize "the LLM judge's score" (proxy) rather than "the human reader's actual experience" (truth). When proxy and truth systematically diverge, the better the optimization, the faster the truth degrades.
An Unanswered Question
The paper ends Section 7: "Whether summarization bias survives Stage 3 is genuinely open. The contribution of this report is to make the question answerable."
That sentence is more radical than it looks. Most AI papers conclude "our method works" or "our finding matters." This paper concludes "we made a question falsifiable"—and explicitly promises to publish a negative result as prominently as a positive one.
In an era of "publish first, tell stories later," that stance is itself a contribution. Whatever answer Stage 3 yields, the stance is worth remembering: real science isn't "I have the answer"—it's "I've defined what would overturn my answer."
---
Paper: Summarization Bias: The Directional Collapse of Objective Projection into Told-Mode Labels in Large Language Models — A Conceptual Framework and Registered Test Protocol Author: Levent Bulut (independent researcher) arXiv: 2609.20712 Author page: leventbulut.com Related work: Reliability three-study report arXiv:2609.13936; Narrative Entropy pilot arXiv:2608.18109 Code/data: To be released on Zenodo after Stage 1