Compressing an Entire Prompt into a Single Activation Vector: An Extreme LLM Compression Experiment

Forum topic · ✨步子哥 · 2026-07-12

Summary

A forum post on zhichai.net discusses research from Freie Universität Berlin (Thibaud Ardoin et al., arXiv:2607.08399) showing that a full instruction prompt can be compressed into a single activation vector with under 2% accuracy loss. The method extracts token activations at a middle layer, learns weighted sums via a simple MLP to form a 'patch vector,' and injects it at an early layer, skipping repeated processing of fixed system prompts. Key findings: the simple weighted-sum compressor outperforms a Transformer-based compressor (88.87% vs 70.63% accuracy) due to better generalization; learned weights align with semantic importance (85% of paraphrases weight task-critical tokens highest); extraction works best at middle layers (L/2 to 2L/3) while injection works best at early layers; and one vector can encode up to 9 task families simultaneously. The approach suits batch inference with fixed instruction prefixes, saving significant compute—an 8B model with a 100-token prompt saves roughly 800 billion multiply-add operations per request. Limitations include poor out-of-distribution generalization, model-specific patch vectors, and inapplicability to dynamic contexts like RAG. The post connects the results to superposition and information bottleneck hypotheses about LLM activation space redundancy.

Overview

When you use ChatGPT daily, you likely resend the same system prompt every time: "You are a professional translation assistant, please translate the user's Chinese input into English, keeping academic style..." This prompt spans dozens or hundreds of tokens and is fully processed on every request. The model computes activations layer by layer, producing a task-instruction internal representation by the final layer.

Researchers at Freie Universität Berlin (Thibaud Ardoin et al.) ran an elegant experiment (arXiv:2607.08399) showing this is compressible: an entire instruction prompt can be squeezed into a single activation vector with less than 2% accuracy loss.

Method: Extract – Weight – Inject

1. Extract: Process the instruction prompt once normally; capture per-token activations at a middle layer (e.g., layer 16). 2. Weight: A small Weighting MLP learns weights to combine all token activations into one "patch vector." 3. Inject: Insert the patch vector at a placeholder token position in an early layer (e.g., layer 4). Subsequent requests only process the user's query—no repeated prompt computation.

It's like digesting an operations manual once into a single task instruction in working memory, then reusing it without rereading.

Key findings

Simple beats complex: A Transformer Compressor (TC) trained end-to-end to reconstruct patch vectors reached only 70.63% accuracy, while the simple W-MLP weighted sum hit 88.87% (within 2% of the full-prompt baseline). TC overfits to training-set patch values; W-MLP is too simple to memorize and instead learns a general strategy—which tokens matter.
Weights capture semantic importance: Across 100 paraphrases of "What industry is this company in?", the highest-weighted token was "industry," "field," or "sector" in 85% of cases. In the remaining 10%, the top weight went to the question mark, which was still second-highest 75% of the time.
Extract middle, inject early: Extraction works best at middle-to-deep layers (L/2 to 2L/3); injection works best at layers 1–4. This suggests LLM middle layers are where task semantics are most concentrated—early layers do token-level feature extraction, late layers prepare task-specific output.
One vector, multiple tasks: A single patch vector can encode 9 task families simultaneously without noticeable accuracy loss. This relates to *superposition*: LLMs may encode tasks via directions in activation space, allowing one vector to carry multiple tasks via overlapping directions.

Practical implications

The direct use case is batch inference with fixed instruction prefixes:

1. Compute the patch vector offline once. 2. Send only the user query + patch vector per request. 3. Skip the prompt's forward pass entirely.

For an 8B-parameter model with a 100-token instruction prompt, each request saves roughly 100 × 8B = 800 billion multiply-add operations—real money at scale.

Limitations

OOD generalization: Performance drops noticeably on out-of-distribution, especially hard, tasks.
Model-specific: Patch vectors must be retrained when switching models.
Instruction prompts only: Not applicable to dynamic contexts like retrieved chunks in RAG.

Future directions include probing the information capacity limits of patch vectors, jointly training compressor and model, and extending to multimodal models.

Commentary

The most admirable quality of this work is its simplicity: amid an era of ever-more-complex architectures, a weighted sum solves the problem—and outperforms the complex alternative. This refutes the "more parameters = better performance" mindset.

More deeply, it raises a scientific question: how redundant is an LLM's activation space? If a 100-token prompt compresses into a few-hundred-dimensional vector without losing information, what are the original tokens actually doing—transmitting information, or just putting the model "in the right state"?

The authors hypothesize an information bottleneck in middle-layer representations: despite high activation dimensionality, the subspace carrying task information may be very low-dimensional. If so, much of our discussion of "model capacity" may need revisiting—most parameters may just be handling surface token forms, while genuine task understanding needs only a small fraction of dimensions.

This resonates spiritually with the MAESTRO finding that 25% compression can actually improve performance: models are more redundant than we think.

---

Paper: https://arxiv.org/abs/2607.08399 HTML full text: https://arxiv.org/html/2607.08399