You have a 2,000-token system prompt that gets re-processed on every user query. KV caching saves some compute, but the prompt tokens still need a full forward pass. At scale, this gets expensive.
A team at Freie Universität Berlin led by Thibaud Ardoin asked a bold question in their July 2026 paper:
Can the information in those 2,000 tokens be compressed into a single vector?
Not 2,000 vectors. Not 200. One.
Then inject that one vector back into some layer of the model so it behaves as if it had just processed the full 2,000 tokens.
The answer: yes, with only about 2% accuracy loss.
---
Method: Weighted Activation Aggregation
The core idea is strikingly simple. As an LLM processes a prompt, each layer produces activation vectors. The approach:
1. Extract: at a chosen middle layer (e.g., layer 16), collect the activation vectors of all prompt tokens 2. Weighted sum: apply a learned weighting function (a Weighting MLP) to combine them into a single "compressed vector" 3. Inject: feed this compressed vector into an early layer (e.g., layer 4) during query processing, replacing the original prompt token sequence
Formally, if the prompt produces activations h_1, h_2, ..., h_n at layer L, the compressed vector is v = Σ w_i · h_i, where w_i comes from the Weighting MLP. This v is injected at layer L' (L' < L), acting as a virtual prompt representation.
Two Compressors
Weighting MLP (W-MLP): a small MLP that takes each token's activation and outputs a scalar weight. Simple and efficient.
Transformer Compressor (TC): a small Transformer that models token interactions — more expressive in theory, but costlier to train.
Experiments show W-MLP is already sufficient: on task-instruction compression it performs on par with TC while being simpler.
Key Experimental Results
Tested on Llama-3:
- Task-instruction compression (Toy Task Dataset): W-MLP accuracy drops less than 2% versus full-prompt processing
- In-distribution tasks: W-MLP stays within 2% of the full-prompt baseline
- Failure cases: when compression fails (the vector fails to encode the information), performance falls to near random guessing — evidence that the vector genuinely carries the information
- Tested only on fixed instruction prompts; dynamic prompts are not covered
- Compression is task-specific — the Weighting MLP must be trained per target task, not plug-and-play
- A 2% loss may be unacceptable in high-stakes domains (medicine, law)
- Generalization unverified beyond Llama-3
- Token-level prompt compression (e.g., LLMLingua) removes redundant tokens but still processes a sequence; activation aggregation skips tokens entirely
- KV cache avoids recomputation but still consumes memory; activation aggregation compresses the whole prompt into one vector
- Activation steering / representation engineering: aggregation can be seen as steering with a compressed prompt
- Superposition hypothesis: weighted summation only works if information is linearly encoded, providing indirect evidence for it
What does 2% mean in practice? A 2,000-token system prompt invoked 1 million times saves 2,000 tokens of forward-pass compute per call, at the cost of 2% accuracy. For many applications, that trade is worth it.
Three Findings About LLM Activation Structure
1. Cross-layer compatibility. Activations from layer 16 work when injected at layer 4. Layers share a kind of "common language" — information encoding is not reinvented per layer. This is counterintuitive given the usual shallow-syntax/deep-semantics division of labor; deep semantic information can apparently be "read" by shallow layers.
2. Information capacity of a single vector. A 2,000-token prompt unfolds into 2,000 vectors of, say, 4,096 dimensions — yet compressing to 1 × 4,096 costs only 2% accuracy. The original prompt must be highly redundant: the task-relevant semantic content fits in one vector.
3. Weighted summation is a robust compressor. It works because tokens differ in importance — task keywords matter far more than punctuation and stop words — and the Weighting MLP learns those weights. This structurally resembles attention, but with static, prompt-only weights computed once and reused across all queries.
Beyond Task Instructions: Multi-Task Encoding
Appendix C explores encoding multiple task instructions into a single vector. Success depends on task separability: sufficiently distinct tasks (e.g., sentiment classification vs. math) can co-exist in one vector; similar tasks interfere. This suggests future "prompt libraries" could be collections of vectors, loaded by injection rather than token re-processing.
Limitations
Relation to Other Methods
Why This Matters More Than It Seems
Beyond inference acceleration, the paper addresses a basic question: how is prompt semantics distributed inside a model? The answer appears to be: highly concentrated. The semantics of 2,000 tokens can be captured in one 4,096-dim vector at ~98% fidelity — all tokens jointly point toward a "semantic centroid" that weighted summation approximates. This echoes findings on Procrustes alignment between independently trained models: LLM internal representations are more organized than commonly assumed.
---
Paper: Prompt Compression via Activation Aggregation Authors: Thibaud Ardoin, Semira Einsele, Evis Bregu, Gerhard Wunder (Freie Universität Berlin) Date: July 9, 2026