A Disastrous Scenario Nobody Noticed
Imagine you are a corporate compliance officer. You feed 76 internal policy documents—each 50 to 300 pages—into an AI agent and ask: which requirements in these documents are mandatory?
The agent confidently lists 20 items. But when you check the originals, half of the truly important requirements are missing, while ordinary sentences the author never emphasized get flagged as "must comply."
This isn't because the agent is dumb—it's running Claude Opus 5 or GPT-5.6. The problem occurs earlier: when the .docx is converted into text the agent can read, key information the author marked with color or highlighting is silently discarded.
This paper quantifies the problem. The numbers are ugly.
Three Reasons "Silent Cue Loss" Is Dangerous
The authors call this failure mode Silent Cue Loss, and it has three features that make it more dangerous than ordinary format loss:
1. Silence: The ingestion pipeline never reports "I dropped some information." No warnings, no error codes, no monitoring triggers. The text the agent receives *looks* complete—the critical markers are just gone.
2. Inverted failure mode: The agent doesn't answer less because it doesn't know—it answers *wrongly* because it doesn't know. Its false positive rate—treating sentences the author never marked as requirements—surges to 4x the correct method. The agent isn't hesitating; it's confidently making things up.
3. Pervasiveness: This is not an edge case. Among 6,460 real .docx policy documents, 32.8% contain deliberate color or highlight marks, and 10.3% carry marks on obligation-expressing sentences. One in three documents is affected—this is the norm for enterprise documents.
The Numbers: From 0.690 to 0.075
The experimental design is straightforward: 76 real policy documents, each with "requirement" sentences marked by the author in red text or highlight, plus unmarked "non-requirement" sentences. Agents must list all requirements under different ingestion methods.
Results (micro F1):
- Method that correctly preserves formatting: 0.690
- docling (mainstream document converter): 0.150
- Plain text: 0.075
- Highlight: +0.302 (p < 1e-4 after Holm correction)
- Color: +0.262 (p < 1e-4)
- Bold: ±0.003 (not significant)
- Italic: ±0.003 (not significant)
- Monospace: ±0.003 (not significant)
- Strikethrough: ±0.003 (not significant)
- Single task type: Only "list requirements" was tested. The importance of color/highlight may differ for other tasks (summarization, QA, information extraction).
- Domain-specific: Policy documents may rely on color marking more than other document types. Marketing docs, technical docs, and legal contracts may carry different format semantics.
- The +0.085 semantic gain is marginal: p = 0.049 barely clears the 0.05 line, and the effect is modest—the authors honestly report this.
From 0.690 to 0.075—a 89% accuracy drop. The gap replicates across six different response models—five Claude models (Opus, Sonnet, Haiku tiers) and OpenAI's GPT-5.6. This is not one model's bug; *every* model collapses once formatting information is lost.
Worse still, docling's false positive rate—flagging sentences the author never marked—is four times that of the correct method. The agent doesn't just "answer less"; it "answers wildly."
Localization: Not All Formatting Matters
The paper ran a fine-grained ablation: leave-one-out analysis per cue category, measuring which formatting cues contribute most to accuracy.
Six cue categories: highlight, color, bold, italic, monospace, strikethrough.
Results:
The entire effect comes almost exclusively from highlight and color cues. Bold, italics, monospace, and strikethrough—formats we assumed "also matter"—contribute nothing to agent comprehension.
This turns the vague intuition "formatting matters" into a concrete, actionable conclusion: you only need to preserve color and highlight; other cues can be discarded without hurting agent performance.
Seeing the Cue Isn't Enough—the Agent Must Know What It Means
The paper also includes a pre-registered experiment: is preserving color cues sufficient, or must you additionally tell the agent "red means important"?
Result: preserving color + explaining its meaning adds another +0.085 over preserving color alone (p = 0.049 after Holm correction).
This means cue loss has two layers: 1. Perceptual loss: the converter discards color, so the agent never sees it 2. Semantic loss: even when the agent sees the color, it won't exploit it without knowing "red = important"
Fixing requires both layers: first preserve perception (let the agent see color), then supply semantics (tell the agent what color means).
Why Mainstream Converters Drop It
Mainstream converters (docling, unstructured, etc.) are designed to "turn documents into clean text." Under that goal, color and highlight are "visual decoration," not "semantic content," and naturally get discarded.
But this design assumption is wrong. In enterprise documents, color and highlight are not decoration—they are explicit semantic signals from the author. Authors mark "must comply" in red and "note the risk" in yellow—these signals matter as much as the text itself.
Worse, no mainstream document parsing benchmark tests color/highlight preservation. Parser benchmarks measure "can it extract text correctly," "can it recognize tables," "can it handle formulas"—but never "can it preserve color." So parsers score well on benchmarks while failing silently in real deployments.
Practical Advice for Agent Builders
The paper's practical guidance is clear:
1. Ingestion is where reliability is won or lost. Don't pour all your effort into reasoning and retrieval. If your ingestion pipeline loses key information, no amount of reasoning can save it. Ingestion, not reasoning, is where reliability is won or lost.
2. The fix is cheap and measurable. Preserving color and highlight requires no complex ML—just read run properties during OOXML parsing and keep color information as structured markup. Extremely low cost, quantifiable effect (F1 from 0.150 to 0.690).
3. Don't trust text that "looks clean." Plain text looks complete, readable, fine. But "looks complete" and "is informationally complete" are two different things. Any pipeline converting .docx or PDF to plain text may silently lose critical semantic signals.
The Bigger Picture: A Blind Spot in Multimodal Agents
The paper points to a broader issue: current LLM agents' "multimodal" capabilities have a blind spot.
When we say "multimodal," we usually mean "can see images, hear audio." But documents are also a modality—they have visual structure (color, highlight, layout, fonts) that carries semantic information. Flattening documents to plain text discards an entire modality.
This aligns with earlier findings about VLMs (vision-language models) in document understanding: VLMs can perceive a document's visual structure, but most current agent frameworks don't use VLMs to read documents—they use text converters + plain-text LLMs. This pipeline systematically fails on visual-semantic signals like color and highlight.
Future agents may need to read documents directly with VLMs—not "convert to text, then understand," but "look at documents the way humans do." Until then, at minimum, preserve color and highlight in your text conversion pipeline.
Honest Assessment
The paper has limitations:
---
Paper: arxiv.org/abs/2610.30026 Code: The authors state it will be open-sourced.