When you use ChatGPT daily, you probably resend the same system prompt every time: "You are a professional translation assistant..." That's dozens or hundreds of tokens, fully processed on every request — layer by layer, until some layer (say layer 32) produces an internal representation containing the task instruction.
But have you ever wondered: how big is the task information carried by that prompt after 32 layers? Could it fit in a single vector that you inject directly next time, skipping those repeated computations?
Researchers at Free University of Berlin (Thibaud Ardoin et al., arXiv:2607.08399) ran an elegant experiment showing it's feasible — compressing an instruction prompt into a single activation vector with less than 2% accuracy loss.
Core method: Extract — Weight — Inject
1. Extract: process the instruction prompt normally once, capturing each token's activation vector at a middle layer (e.g., layer 16). 2. Weight: a small Weighting MLP learns a set of weights to compute a weighted sum over token activations, producing a single "patch vector." 3. Inject: inject the patch vector into a placeholder token position at an early layer (e.g., layer 4). User queries are then processed normally — the instruction prompt is never re-processed.
It's like reading an operations manual once, digesting it into a single task instruction in working memory, and reusing it instead of re-reading the manual every time.
Finding 1: Simple beats complex
The paper compares two compressors:
- W-MLP: a simple weighted sum with MLP-learned weights.
- TC (Transformer Compressor): a small Transformer trained end-to-end to reconstruct the patch vector.
- Extraction: mid-to-deep layers (around L/2 to 2L/3) work best. Too shallow lacks abstract task representations; too deep binds to specific token positions and generalizes poorly.
- Injection: the earlier the better (layers 1–4), giving the patch vector more layers to fuse with the query.
- OOD generalization: performance drops notably on out-of-distribution, especially hard, tasks.
- Model-specific: patch vectors require retraining the compressor for each model.
- Instruction prompts only: not applicable to dynamic contexts (e.g., RAG passages).
Counterintuitively, W-MLP stays within 2% of the full-prompt baseline (88.87% accuracy), while TC only reaches 70.63%. TC overfits to the specific patch values of the training set; W-MLP is too simple to memorize, so it learns something general — which tokens matter and which don't — consistent with the classic "simple models generalize better" intuition.
Finding 2: Learned weights capture semantic importance
The authors generated 100 paraphrases of the prompt "What industry is this company in?" and examined the learned weight distributions. In 85% of paraphrases, the highest weight went to "industry", "field", or "sector" — the core semantic target. In the remaining 10%, the top weight went to the question mark, which was still second-highest in 75% of cases. W-MLP isn't memorizing token positions; it genuinely identifies which word carries the task's core semantics.
Finding 3: Extract from middle layers, inject at early layers
Systematic tests over extraction/injection layer pairs reveal a consistent pattern:
This suggests the middle layers of an LLM are where "task semantics" are most condensed — early layers do token-level feature extraction, late layers prepare task-specific outputs, and the middle holds the abstract "what is this task?" representation.
Finding 4: One vector can hold multiple tasks
Surprisingly, a single patch vector can encode up to 9 task families simultaneously without notable accuracy loss. A few-hundred-dimensional vector can distinguish translation, summarization, QA, and classification at once. The authors connect this to superposition: LLMs may encode tasks by directions rather than positions, so one vector can carry multiple tasks via overlapping directions.
Practical significance
The direct application is batch inference with fixed instruction prefixes:
1. Compute the patch vector offline once. 2. Send only the user query + patch vector per request. 3. Skip the instruction prompt's forward pass entirely.
For an 8B-parameter model with a 100-token prompt, that saves roughly 100 × 8B = 800 billion multiply-adds per request — real money at scale.
Limitations and outlook
My take
What I admire most is the simplicity — a weighted sum beats a more complex architecture, a nice rebuttal to "more parameters = better performance."
More deeply, this touches a fascinating scientific question: how redundant is an LLM's activation space? If a 100-token prompt fits into a few hundred dimensions without information loss, what are the other tokens doing — transmitting information, or just getting the model "into the right state"?
The authors boldly conjecture an information bottleneck in mid-layer representations: the task-relevant subspace may be very low-dimensional. If so, much of our discussion of "model capacity" may need revisiting — most parameters may be handling surface forms, while true task understanding needs only a small subspace. This resonates with the MAESTRO finding that 25% compression can actually improve performance: models are more redundant than we think.
---
Paper: https://arxiv.org/abs/2607.08399 HTML full text: https://arxiv.org/html/2607.08399