You have a 2000-token system prompt that gets reprocessed on every user query. KV cache saves some computation, but the prompt's tokens still need a full forward pass. At scale, this gets expensive.
A team at Freie Universität Berlin led by Thibaud Ardoin (July 2026 paper) asked a bold question:
Can the information in those 2000 tokens be compressed into one vector?
Not 2000 vectors. Not 200. One.
Then inject that single vector back into one of the model's layers so the model behaves as if it had just processed those 2000 tokens.
The answer: yes, with only ~2% accuracy loss.
Method: Weighted Activation Aggregation
The core idea is surprisingly simple. As an LLM processes a prompt, each layer produces activation vectors. The approach:
1. Extract: At a middle layer (e.g., layer 16), collect the activation vectors of all prompt tokens 2. Weighted sum: Apply a learned weighting function (a Weighting MLP) to aggregate them into a single "compressed vector" 3. Inject: Inject this compressed vector into an early layer (e.g., layer 4) of a query's forward pass, replacing the original prompt token sequence
Formally: if the prompt produces activations h_1, h_2, ..., h_n at layer L, the compressed vector is v = Σ w_i · h_i, where w_i comes from the Weighting MLP. This v is injected at layer L' (L' < L) of the query, like inserting a virtual prompt representation.
Two Compressors
- Weighting MLP (W-MLP): a small MLP mapping each token's activation to a scalar weight. Simple and efficient.
- Transformer Compressor (TC): a small Transformer that models token interactions; more expressive but costlier to train.
- Task instruction compression (Toy Task Dataset): W-MLP accuracy is within 2% of full-prompt processing
- Train/test: on in-distribution tasks, gap to the full-prompt baseline is < 2%
- Failure case: when compression fails (the vector fails to encode the information), performance drops to near chance—evidence the vector genuinely carries information
- Fixed instruction prompts only: dynamic, input-dependent prompts don't apply
- Task-specific: the Weighting MLP must be trained per target task; not plug-and-play
- 2% may be too much for high-precision domains (medical, legal)
- Generalization unverified: tested only on Llama-3
- Token-level prompt compression (e.g., LLMLingua): deletes redundant tokens but still processes a sequence; activation aggregation skips the token level entirely
- KV cache: avoids recomputation but still occupies memory; activation aggregation compresses the whole prompt into one vector
- Activation steering / representation engineering: modifying activations to steer behavior—aggregation can be seen as steering with a compressed prompt
- Superposition hypothesis: if features weren't linearly encoded, weighted summation couldn't work—single-vector compression is indirect evidence for linear feature encoding
Experiments show W-MLP is already sufficient—for task instruction compression it matches TC while being simpler and cheaper.
Key Results (Llama-3)
Scale example: a 2000-token system prompt called 1 million times saves a 2000-token forward pass per call, at a 2% accuracy cost. For many applications, that trade is worth it.
Three Findings About LLM Activation Geometry
1. Cross-layer compatibility. Middle-layer (16) activations work when injected at an early layer (4). LLM layers appear to share a "common language"—information encoding is not reinvented per layer. This is counterintuitive given the usual shallow-syntax/deep-semantics picture, but it suggests deep semantic information can be "read" by shallow layers: layer functions are not strictly partitioned.
2. Information capacity of a single vector. A 2000-token prompt yields 2000 vectors of, say, 4096 dimensions—compressing to 1 × 4096 discards 99.95% of that "capacity" in information-theoretic terms. Yet accuracy drops only 2%. The conclusion: prompts are highly redundant—the task-relevant information fits comfortably in one 4096-dim vector.
3. Weighted sum is a robust compressor. Tokens differ in importance—task keywords matter far more than punctuation and stopwords. The Weighting MLP learns which tokens matter. This is structurally similar to attention (both aggregate tokens with weights), except attention uses query-dependent dynamic weights while this is a static, prompt-only weighting computed once and reused across all queries.
Beyond Single Tasks: Multi-Task Encoding
Appendix C explores whether multiple task instructions can be encoded into one vector. Results depend on task separation: sufficiently distinct tasks (e.g., sentiment classification vs. math) coexist in one vector; similar tasks interfere.
This hints at a future where a "prompt library" is a set of vectors—loading a task means injecting one vector, not reprocessing tokens.
Limitations
Relation to Other Work
Why This Matters More Than It Looks
Beyond inference acceleration, the paper addresses a fundamental question: how is a prompt's semantic information distributed inside the model?
The answer appears to be: highly concentrated. The semantics of 2000 tokens fit into one 4096-dim vector at ~98% fidelity. Tokens don't each contribute independent information; they collectively point to a "semantic center of mass" that a weighted sum approximates.
This echoes findings from Procrustes alignment work (two independently trained models aligning at r=0.70): LLM internal representations are far more organized than we assumed.
---
Paper: Prompt Compression via Activation Aggregation Authors: Thibaud Ardoin, Semira Einsele, Evis Bregu, Gerhard Wunder (Freie Universität Berlin) Date: July 9, 2026