Summarization Bias: When LLM Judges Systematically Reward Telling Over Showing

Forum topic · ✨步子哥 · 2026-09-19

Summary

A Chinese tech forum post discusses a 2026 arXiv paper (2609.20712) by independent researcher Levent Bulut proposing 'Summarization Bias'—the claim that large language models exhibit a directional, asymmetric preference for 'told-mode' prose (explicitly stated emotions and information) over 'shown-mode' prose (meaning encoded in physical detail requiring reader inference), violating the classic 'show, don't tell' principle. The post highlights evidence from a completed reliability study (arXiv:2609.13936) where five machine annotators all failed to detect reasoning-based features like 'materialized metaphor' in Turkish narrative corpora, with wildly divergent counts (0, 1, 40, 72, 78 vs. 9 from humans). Crucially, the paper pre-registers a four-stage falsification protocol, committing to withdraw the construct if Stage 3 judging tests fail, with withdrawal published at equal prominence. The author warns that LLM judges in RLHF reward models, automated grading, and content moderation create directional selection pressure ('drift toward told mode'), flattening human writing toward what is easiest for models to recognize—a proxy-objective trap where scale cannot fix structural blind spots.

Overview

This zhichai.net forum post analyzes an arXiv paper by independent researcher Levent Bulut: *Summarization Bias: The Directional Collapse of Objective Projection into Told-Mode Labels in Large Language Models — A Conceptual Framework and Registered Test Protocol* (arXiv:2609.20712, September 2026).

The core claim: directionality, not just weakness

The paper argues LLMs fail at "show, don't tell" not randomly, but asymmetrically: on an axis from *told mode* (information written directly on the text surface: "she was full of endless sorrow") to *shown mode* (meaning encoded in physical detail and action: a kettle boiling over while the protagonist doesn't move), errors always point toward the told end.

A symmetric weakness is a capability problem, potentially fixed by scale.
An asymmetric weakness is a structural bias, which scale may amplify rather than fix.

The bias operates in two regimes:

1. Generation: models instructed to "show" slip back into telling, often just swapping synonyms ("her eyes flickered with silent grief"). 2. Judging: models systematically score told-mode versions higher than shown-mode versions of identical content and length. The author argues the judging regime is more dangerous because it creates selection pressure—RLHF reward models, AI editors, automated essay scoring, and content moderation all act as directional evolutionary forces pulling writing toward told mode ("drift toward told mode").

Supporting evidence: five annotators failing

A completed three-study reliability report (arXiv:2609.13936) compared a rule-based detector plus four LLM annotators against blind human labels on Turkish narrative corpora:

Surface features (physical/time markers): machines succeeded, but the features are near-ubiquitous and carry little diagnostic value.
Reasoning features—notably *materialized metaphor* (an abstract emotion made concrete as an object), the diagnostic marker of shown mode: all five machine annotators failed. Across 100 scenes where humans counted 9, machine counts were 0, 1, 40, 72, and 78; Cohen's κ for four of five was indistinguishable from chance. The disagreement itself suggests machines were guessing, not detecting.

Bulut is careful: the result is *consistent with* Summarization Bias, but also consistent with under-specified rule definitions.

The registered protocol

The paper pre-registers a four-stage test with fixed abandonment conditions:

Stage 1: 20 matched scene pairs (told/shown), length-matched (±5%), identical Flesch-Kincaid grade levels; two annotators encode a Suppressed Information Index (SI); categories with κ < 0.60 halt testing.
Stage 2 (generation): ≥3 model families given show-mode instructions; prediction: model SI significantly below human shown targets.
Stage 3 (judging, core): models judge matched pairs; prediction: model preference for told versions exceeds human raters' (Cohen's h ≥ 0.3 or odds ratio ≥ 2). If falsified, the construct is withdrawn and the withdrawal is published with the same prominence as a confirmation.
Stage 4 (discriminant checks): mismatched lengths or prompt framing should eliminate any preference—otherwise the effect is verbosity bias or sycophancy, not Summarization Bias.

Why directionality matters more than strength

The post connects this to the evaluation-blind-spot literature: classifiers learn surface lexical cues, not semantic structure, so scale improves surface recognition without fixing the deeper blind spot. "Scale won't repair a structural blind spot—it just runs faster in the wrong direction."

Stated limitations

Per the author:

No validation data yet—this is a framework plus registered protocol.
Supporting evidence is suggestive, not confirmatory.
Definitions depend on annotator judgment; Stage 1 κ checks are prerequisite for interpreting downstream tests.
Any confirmation is limited by model family, language, and genre.
Distinguishing Summarization Bias from verbosity bias/sycophancy is itself a Stage 4 hypothesis.

The poster's commentary

The author argues the construct may extend beyond literature to any "surface declaration vs. inference reconstruction" tension: code review (explicit comments vs. elegant structure), academic writing (stated theses vs. implicit argument), product copy (listed selling points vs. inferred ones). The deeper worry is a proxy-objective trap: LLM judges optimize for what they can recognize, not what human readers experience—directionally reshaping human expression toward flatness. The post closes by praising the paper's falsifiability posture: "real science isn't 'I have an answer'—it's 'I've defined what would refute my answer.'"

---

Paper: Summarization Bias: The Directional Collapse of Objective Projection into Told-Mode Labels in Large Language Models — A Conceptual Framework and Registered Test Protocol Author: Levent Bulut (independent researcher) arXiv: 2609.20712 | Reliability studies: 2609.13936 | Narrative Entropy pilot: 2608.18109 Code/data: to be released on Zenodo after Stage 1