Where Should a Document Live: Context, Representations, or Parameters?

> Paper: *Where Should a Document Live: Context, Representations, or Parameters?* > arXiv: 2609.17346 (Amazon AGI)

A Practical Choice

Suppose you run a customer-service bot that needs to "remember" 10,000 product documents. You have three options:

1. Stuff it into context: prepend relevant documents to the prompt each time (RAG) 2. Compress into representations: encode each document as a KV-cache prefix (Cartridge/Compaction), stitched together at inference 3. Train it into parameters: use LoRA or full fine-tuning to "weld" the knowledge into weights

Each approach has trade-offs, and the industry has argued about it for a long time. Researchers at Amazon AGI did something simple but essential: compared all three head-to-head on the same model, the same data, and the same evaluations.

The result was surprising — there is no winner, only three different failure modes.

Experimental Design: 5 Datasets × 5 Methods × Multiple Compression Ratios

The authors used Qwen3-8B as the base model and tested on five datasets:

• LongHealth: clinical patient records (multiple choice, long documents, 11.7K tokens)
• QASPER: academic paper QA (extractive, 4.7K tokens)
• QuALITY: fiction/nonfiction reading comprehension (multiple choice, 5.7K tokens)
• FinQA: financial-report numerical reasoning (math, 1K tokens)
• TechQA: IBM IT support documents (extractive, 1.5K tokens)

Methods covered:

• ICL (context): documents concatenated into the prompt
• Cartridge (representation-KV): documents encoded as KV-cache prefixes
• Compaction (representation-KV): another KV compression scheme
• LoRA (parameter-low-rank): low-rank adapters
• MLP adapter (parameter-MLP): MLP adapters
• Full fine-tuning (parameter-full): full FT as an upper bound

Each method was swept across 2x, 10x, 20x, 50x, and 100x compression ratios to trace the performance-cost curve.

Finding 1: For Single Documents, Cartridge Wins on Cost-Effectiveness

In single-document settings (questions about one document), Cartridge (KV-cache prefixes) achieves the highest performance at the smallest size on most datasets.

Take LongHealth as an example:

• Cartridge at 2x compression: 83.2 points, 54 MiB
• LoRA r=50: similar performance requires 509 MiB
• Full fine-tuning: slightly higher, but 3072 MiB

Cartridge matches LoRA with 1/10 the size and reaches 90% of Full FT with 1/50 the size — a huge cost saving for industrial deployment.

But Cartridge is not invincible. On FinQA (numerical reasoning), Cartridge 2x only scores 50.7 while Full FT exceeds 70. Numerical reasoning seems to require deeper parameter-level integration.

Finding 2: Multi-Document Composition Breaks Everything

The real surprise comes in multi-document scenarios. The researchers combined k documents' representations/parameters (KV concatenation / weight averaging) and measured performance at k = 1, 3, 5, 10.

Cartridge collapses at k=3. On FinQA:

• k=1: 34.5 points
• k=3: 4.9 points (collapse)
• k=10: near random

LoRA and MLP adapters fare no better. Weight merging (model merging) collapses on all datasets — performance drops to near random at k=3.

The only thing that composes is Joint Training: training one adapter on all k documents together. But that loses the ability to retrieve on demand — you must know in advance which k documents the user will ask about.

This means: current knowledge-injection methods are fundamentally "single-document optimized." Multi-document composition remains an unsolved problem.

Finding 3: Parameter-Level Injection Forgets

The authors also measured "forgetting" — performance on general benchmarks (GSM8K, HumanEval, IFEval, MMLU) after knowledge injection:

• Cartridge/Compaction (representation): almost no forgetting, since model weights are untouched
• LoRA (low-rank): mild forgetting; MMLU drops 1-2 points at r=8
• MLP adapter: severe forgetting; GSM8K drops from 80 to 24 at bottleneck=192
• Full fine-tuning: catastrophic forgetting; 10+ point drops on all benchmarks

Here is a counterintuitive finding: LoRA's "low-rank constraint" is itself a form of regularization. At equal parameter counts, LoRA forgets less than MLP adapters. It is not parameter count that determines forgetting — it is the constraint structure.

Quote from the paper:

> "The low-rank constraint, rather than the number of parameters alone, as what accounts for preserving general capability."

Finding 4: Compression Is Almost Harmless for Cartridge

From 2x to 100x compression, Cartridge on LongHealth only drops from 83.2 to 76.5. A 50x compression for an 8% performance loss — remarkable robustness.

FinQA is the exception: from 50.7 down to 22.1. Numerical reasoning is highly sensitive to KV details and collapses under compression.

This contrast shows that the loss from KV-cache compression is not uniform. Factual QA can withstand compression; numerical reasoning cannot. A plausible explanation: numerical reasoning requires precise positional encoding of numbers, which KV compression blurs.

My Take: The "Three-Body Problem" of Knowledge Injection

This paper suggests a deeper framework. Knowledge injection involves a three-way trade-off:

1. Accuracy: how completely the knowledge is preserved 2. Composability: whether multiple pieces of knowledge can be activated together 3. Persistence: whether new knowledge wipes out old capabilities

Each method occupies a different "corner":

| Method | Accuracy | Composability | Persistence | |--------|----------|---------------|-------------| | Context (ICL) | High | High | Perfect | | Representation (Cartridge) | Medium | Low | Perfect | | Parameters (LoRA/FT) | High | Very low | Poor |

ICL is optimal on all three dimensions — except one: cost. ICL must reprocess all documents on every inference, and token costs explode. The whole point of Cartridge and LoRA is to trade "composability" and "persistence" for "cost."

But the paper reveals a harsh truth: this exchange is more expensive than expected. Cartridge loses composability the moment multiple documents enter the picture; LoRA loses persistence once tasks multiply.

In other words: all current "knowledge injection" methods are essentially trading away some dimension of ICL for cost. There is no free lunch.

An Overlooked Insight: Is Joint Training the Only Way Out?

The most thought-provoking finding in the paper is that Joint Training significantly outperforms Merging (weight averaging) in multi-document settings.

At k=3, Joint Training scores 13.0 on FinQA versus 4.9 for Merging; the gap widens at k=5.

But Joint Training has a fatal limitation: you must know in advance which documents the user will ask about. This breaks the on-demand-retrieval RAG paradigm.

This leads to a hypothesis: perhaps "knowledge injection" is itself a false premise. Real knowledge — the kind that needs composition and cross-document reasoning — may only be able to live in context. Parameters and representations suit "skills" and "styles," but not "facts."

The paper never states this conclusion explicitly, but the data points that way.

Limitations

1. Only Qwen3-8B tested: other architectures (MoE, encoder-decoder) may behave differently 2. QA-heavy datasets: generation and dialogue tasks untested 3. Cartridge training cost underexplored: inference is cheap, but encoding each document requires a forward pass 4. Only concatenation/averaging tested for composition: more complex composition (e.g., attention-level routing) unexplored

Conclusion

The value of this paper lies not in giving an answer but in laying out the problem. Knowledge injection is not a question of "which method is best" but of "which dimension are you willing to sacrifice."

The next time someone tells you "RAG is dead, fine-tuning is the future" or "Cartridge crushes RAG," show them this paper. The data will tell them: there is no silver bullet, only trade-offs.

---

Paper: https://arxiv.org/abs/2609.17346 HTML version: https://arxiv.org/html/2609.17346v1