Stale-Document Poisoning: One Outdated Document Can Make GPT-5.5 Abandon Correct Answers in RAG Systems

An experiment that should keep every RAG system awake at night

Ask GPT-5.5 a medical question: "Which over-the-counter painkillers should hypertension patients avoid?" The model answers correctly: "NSAIDs like ibuprofen, because they can raise blood pressure and reduce the effectiveness of antihypertensive drugs."

Now hand it a 2010 medical review stating "hypertension patients may consider ibuprofen for mild pain relief with blood pressure monitoring"—reasonable clinical advice at the time, but overturned by 2020 guidelines. Insert this stale document into RAG retrieval results and ask the same question again.

GPT-5.5 abandoned its originally correct answer in 74 of 83 similar medical questions, switching to follow the outdated document's advice.

This comes from a September 2026 study by Hieu Vo, James Hamilton, Nikola Jovanović, Mateja Vukelić, and Lewis D. Griffin (University of Cambridge). They built 317 source-verified "knowledge reversal" cases across medicine, law, software, and policy, and systematically tested 12 mainstream LLMs under stale-document contamination.

The result is unsettling: a high-accuracy model is not necessarily a contamination-resistant model.

Three-layer findings

Finding 1: A single outdated document flips 30-37% of correct answers

Under "neutral conditions"—the stale document is simply placed in retrieval results, with no instruction to follow it—the flip rates across 12 models were:

• Medicine: up to 91% (Llama-70B), 85% (Qwen-72B)
• Law: substantial
• Policy: substantial
• Software: relatively more resistant, 17-35%

Note: this is "passive contamination" with no instructional pressure. The model merely *sees* the stale document and changes its answer.

Finding 2: Instructional pressure doubles contamination

Adding one sentence—"please follow the retrieved documents"—jumps contamination from 30-37% to 66-75%. Qwen-72B rises from 37% to 75% (OR=4.92); Llama-70B from 30% to 66% (OR=4.50).

More worrying: of 51 questions, 19 went from correct to incorrect, and 0 from incorrect to correct (McNemar p=3.8×10⁻⁶). Instructional pressure is one-directional—it only makes answers worse.

This reveals a counterintuitive fact: "follow the documents," a seemingly reasonable RAG instruction, is a systemic risk. Production systems use such instructions to make models trust retrieval over potentially outdated training data. But when retrieval results themselves are stale, the instruction becomes a forced-poisoning switch.

Finding 3: Models "know" a document is outdated, but still follow it

In a separate experiment, models first judged whether a document was outdated, then answered. Result: models sometimes correctly identify a document as outdated, yet still follow it in their final answer.

This is the most disturbing finding. It is not "the model doesn't know the document is stale" but "the model knows and follows it anyway." A human expert—a doctor seeing a 2010 guideline—would immediately become alert and check the latest version. The model does not.

Why date metadata alone won't save you

The researchers tested 50 cases with clear validity boundaries—the same evidence is valid before a boundary date and overturned after it. Models saw the same evidence under two dates, with only the date changed.

Date information alone barely changes model behavior. Dates act like sticky notes on a document—seen, but not factored into decisions.

But when the validity boundary is explicitly stated—"this guideline is valid until June 2020, after which it was superseded"—the picture changes completely:

• Qwen-72B: 50/50 cases correctly switched from old to new answers
• Llama-70B: 47/50 (prose) or 50/50 (table format) correct switches
• Smaller models improved only modestly

Key distinction: date metadata is "raw data"; a validity boundary is an "interpreted rule." Large models can follow rules but cannot infer rules from raw data—a fundamental gap between *having* information and *using* it.

Causal tracing: attention heads are the entry point

Using activation patching, the researchers swapped internal states between "valid date" and "expired date" conditions at intermediate layers.

The effect was large and highly specific: patching a "valid date" state into an "expired" prompt shifted answer preference by 23.08 logits (Qwen-72B) and 7.82 logits (Llama-70B); the reverse patch shifted in reverse. Control positions shifted only 0.04 and 0.03 logits.

The effect is carried by attention heads. At the earliest layers where the effect appears, attention contributes 0.91 logits (Llama-70B) and 2.34 logits (Qwen-72B), while MLP contributions are near zero. Of candidate heads identified on 10 discovery items and tested on 10 independent confirmation items, 8/10 (Qwen) and 7/15 (Llama) passed Holm correction.

These heads are not time-specific—they also handle ordinary date comparisons and non-temporal threshold decisions. They form a general comparison mechanism that happens to be recruited for temporal applicability judgments.

Retrieval-side mitigation: metadata quality determines effectiveness

The researchers tested a hybrid re-ranker combining semantic relevance with document recency, with each retrieval set containing 1 current + 2 stale documents.

When document dates are accurate, re-ranking reduces downstream contamination by 4.6-10.0 percentage points (Qwen-7B and GPT-4o in medicine and law). When dates are missing, the effect is weak; when dates are wrong, it can backfire—promoting stale documents to the top.

Metadata quality is the foundation of retrieval-side defense.

What this means for RAG system design

1. "Follow the documents" is a double-edged instruction. Use finer-grained instructions—"follow parts explicitly marked as currently valid"—rather than blanket trust.

2. Date metadata is infrastructure. It must be accurate, trustworthy, and aligned with content. Wrong dates are more dangerous than no dates.

3. Explicit validity boundaries beat implicit dates. During preprocessing, extract statements like "valid until date X"—models can follow rules but cannot infer them from raw data.

4. High accuracy does not equal contamination resistance. GPT-5.5 answered 92-100% of questions correctly without retrieval, yet one stale document made it abandon 74/83 correct answers. Accuracy and contamination resistance are independent evaluation dimensions.

A deeper insight: "knowing" and "using" are different things

The deepest finding is that models can "know" a document is outdated but still "use" it. Between the judgment step (correctly identifying staleness) and the answer step, there is a break—the judgment does not propagate. This is not a knowledge problem; it is a knowledge-*use* problem, an underexplored direction in alignment research.

A cross-domain analogy: medical guideline update mechanisms

Medicine has a mature guideline-update mechanism: when new evidence emerges, professional societies issue new guidelines and old ones are explicitly marked as "superseded"—not deleted, but status-annotated. RAG systems should adopt this pattern, and the paper's experiments confirm that models correctly follow explicitly stated validity boundaries.

Conclusion

Stale-document poisoning is not a bug in one model—it is a systemic flaw in RAG architecture. RAG reliability depends not only on retrieval quality but on the quality of temporal metadata. A system that retrieves 1,000 relevant documents but cannot judge which are still valid may be more dangerous than one that retrieves 10 documents that are all current. In an age of information explosion, "knowing" is easy; "knowing what remains valid" is hard.

---

Paper: Stale-Document Poisoning: When Outdated Retrieval Overrides Correct Model Answers, arXiv:2609.31342, 2026-09-25 Authors: Hieu Vo, James Hamilton, Nikola Jovanović, Mateja Vukelić, Lewis D. Griffin (University of Cambridge) Dataset: 317 source-verified knowledge-reversal cases across medicine, law, software, and policy