Where Should a Document Live: Context, Representations, or Parameters?

> Paper: *Where Should a Document Live: Context, Representations, or Parameters?* > arXiv: 2609.17346 (Amazon AGI)

The Practical Choice

Suppose you run a customer-service bot that must "remember" 10,000 product documents. You have three options:

1. Stuff the context: retrieve and prepend relevant documents to the prompt (RAG) 2. Compress into representations: encode each document into a KV-cache prefix (Cartridge/Compaction) and concatenate at inference 3. Train into parameters: bake the knowledge into weights via LoRA or full fine-tuning

Each approach has tradeoffs, and the debate has raged in industry. Amazon AGI researchers did something simple but essential: compared all three head-to-head on the same model, the same data, and the same evaluation.

The result was surprising — there is no winner, only three distinct failure modes.

Experimental Setup: 5 Datasets × 5+ Methods × Multiple Compression Ratios

Base model: Qwen3-8B, evaluated on five datasets:

• LongHealth: clinical patient records (multiple choice, long documents ~11.7K tokens)
• QASPER: academic paper QA (extractive, ~4.7K tokens)
• QuALITY: fiction/non-fiction reading comprehension (multiple choice, ~5.7K tokens)
• FinQA: financial report numerical reasoning (math, ~1K tokens)
• TechQA: IBM IT support documents (extractive, ~1.5K tokens)

Methods covered:

• ICL (context): documents concatenated into the prompt
• Cartridge (representation-KV): documents encoded as a KV-cache prefix
• Compaction (representation-KV): another KV compression scheme
• LoRA (parameter, low-rank)
• MLP adapter (parameter)
• Full fine-tuning (parameter) as an upper bound

Each method was swept across compression ratios of 2x, 10x, 20x, 50x, and 100x to trace performance-cost curves.

Finding 1: In Single-Document Settings, Cartridge Wins on Cost-Effectiveness

For single-document tasks, Cartridge (KV-cache prefix) achieves the highest performance at the smallest size on most datasets.

Example from LongHealth:

• Cartridge at 2x compression: 83.2, using 54 MiB
• LoRA r=50: similar performance but 509 MiB
• Full fine-tuning: slightly higher, but 3072 MiB

Cartridge matches LoRA at ~1/10 the size and reaches ~90% of full fine-tuning at ~1/50 the size — a massive industrial cost saving.

But Cartridge is not invincible. On FinQA (numerical reasoning), Cartridge at 2x scores only 50.7 while full fine-tuning exceeds 70. Numerical reasoning appears to require deeper parameter-level integration.

Finding 2: Multi-Document Composition Breaks Everything

The real surprise comes from multi-document scenarios, where k documents' representations/parameters are combined (KV concatenation / weight averaging) and performance is tracked at k = 1, 3, 5, 10.

Cartridge collapses at k=3. On FinQA:

• k=1: 34.5
• k=3: 4.9 (collapse)
• k=10: near random

LoRA and MLP fare no better. Weight merging (model merging) collapses on all datasets — near-random at k=3.

The only method that composes is joint training: training k documents together into a single adapter. But that forfeits on-demand retrieval — you must know in advance which k documents users will query.

This means current knowledge-injection methods are all fundamentally "single-document" methods. Multi-document composition remains an open problem.

Finding 3: Parameter-Level Injection Causes Forgetting

The authors also measured forgetting: after injecting knowledge, performance on general benchmarks (GSM8K, HumanEval, IFEval, MMLU).

• Cartridge/Compaction (representations): almost no forgetting, since weights are untouched
• LoRA: mild forgetting — MMLU drops 1–2 points at r=8
• MLP adapter: severe forgetting — GSM8K drops from 80 to 24 at bottleneck=192
• Full fine-tuning: catastrophic forgetting — all benchmarks drop 10+ points

Counter-intuitive insight: LoRA's low-rank constraint is itself a regularizer. At comparable parameter counts, LoRA forgets less than MLP adapters. It is not parameter count that determines forgetting, but constraint structure.

> "The low-rank constraint, rather than the number of parameters alone, as what accounts for preserving general capability."

Finding 4: Compression Is Nearly Harmless for Cartridge

Cartridge compressed from 2x to 100x on LongHealth drops only from 83.2 to 76.5 — 50x compression for an 8% performance loss. Remarkable robustness.

But FinQA is the exception: 50.7 drops to 22.1. Numerical reasoning is highly sensitive to KV details, and compression destroys it.

This shows that KV-cache compression losses are not uniform: factual QA tolerates compression; numerical reasoning does not. A plausible cause is that numerical reasoning needs precise positional encoding of numbers, which KV compression blurs.

Reflection: Knowledge Injection's "Three-Body Problem"

A deeper framing: knowledge injection trades off three dimensions:

1. Fidelity: how completely knowledge is preserved 2. Composability: whether multiple knowledge pieces can be activated together 3. Persistence: whether new knowledge erodes existing capabilities

Each method occupies one corner:

| Method | Fidelity | Composability | Persistence | |--------|----------|---------------|-------------| | Context (ICL) | High | High | Perfect | | Representation (Cartridge) | Medium | Low | Perfect | | Parameter (LoRA/FT) | High | Very low | Poor |

ICL is optimal on all three dimensions — except one: cost. ICL re-processes all documents on every inference, exploding token costs. Cartridge and LoRA exist precisely to trade composability and persistence for cost.

But the paper reveals a harsh truth: that exchange is more expensive than assumed. Cartridge collapses compositionally in multi-document settings; LoRA collapses in persistence under multi-tasking. In other words, all current knowledge-injection methods are essentially trading away some dimension of ICL in exchange for cost. No free lunch.

An Overlooked Insight: Is Joint Training the Only Way Out?

The most telling result: joint training significantly outperforms merging on multi-document tasks — 13.0 vs 4.9 on FinQA at k=3, with the gap widening at k=5.

But joint training has a fatal limitation: you must know in advance which documents will be queried, breaking the on-demand RAG paradigm.

This suggests a hypothesis: perhaps "knowledge injection" is a false premise. True knowledge — knowledge requiring composition and cross-document reasoning — may only live in context. Parameters and representations suit "skills" and "styles", not "facts". The paper doesn't state this explicitly, but the data points that way.

Limitations

1. Only Qwen3-8B tested: other architectures (MoE, encoder-decoder) may behave differently 2. QA-heavy datasets: generation and dialogue tasks untested 3. Cartridge training cost under-discussed: encoding each document still requires a forward pass 4. Only concatenation/averaging tested for composition: more complex routing (e.g., attention-level) unexplored

Conclusion

The paper's value lies not in giving an answer but in laying the problem open. Knowledge injection is not about which method is best, but about which dimension you are willing to sacrifice.

Next time someone claims "RAG is dead, fine-tuning is the future" or "Cartridge crushes RAG", show them this paper. The data says: no silver bullets, only tradeoffs.

---

Paper: https://arxiv.org/abs/2609.17346 HTML version: https://arxiv.org/html/2609.17346v1