Summary
Researchers at MIT have proposed AM-OMP (Attention Matching - Orthogonal Matching Pursuit), a training-free method for compressing KV caches, detailed in the paper 'Fast KV Compaction via Attention Matching'. The method rests on an attention mixing identity—the final output of concatenated attention blocks is a weighted mixture of local attention outputs—and adds per-token scalar bias compensation (β) solved in closed form via NNLS. The pipeline has three closed-form steps with no gradient descent: OMP-based greedy key selection, NNLS bias fitting to match attention mass, and OLS value reconstruction to match attention outputs, plus non-uniform per-head compression budgets. On Qwen3-4B, at 50x compression AM-OMP reaches 0.67 accuracy on QuALITY (vs 0.72 uncompressed), compresses a document in about 30 seconds versus roughly 5 GPU-hours for Cartridges, and scores about 0.70 on LongHealth with 60k tokens at 10x compression; combined with summarization it achieves up to 200x total compression. In online compaction on AIME 2025, a physical length of 2048 with effective length 8192 scored 13/30, matching standard 8192 decoding. Paper: arXiv:2602.16284.
MIT's latest AM-OMP (Attention Matching - Orthogonal Matching Pursuit) technique enables extremely fast KV cache compression, presented in the paper *Fast KV Compaction via Attention Matching*.
Core Technical Architecture
1. Theoretical foundation — attention mixing identity: the final output of concatenated attention blocks is a weighted mixture of each local attention output.
2. Scalar bias compensation: introduces a per-token scalar bias β, solved in closed form via NNLS.
3. Three-step closed-form solution (no gradient descent required):
- Key selection: greedy selection via OMP
- Bias fitting: NNLS to match attention mass
- Value reconstruction: OLS to match attention outputs
4. Non-uniform head budget allocation: different attention heads receive different compression ratios.Performance (Qwen3-4B)
- QuALITY accuracy (50x compression): 0.67 (baseline 0.72)
- Compression time: ~30 seconds per document (vs ~5 GPU-hours for Cartridges)
- LongHealth, 60k tokens at 10x compression: ~0.70 accuracy
- Combined with summarization, up to 200x total compression ratio
Online Compaction Experiment (AIME 2025)
With a physical length of 2048 and an effective length of 8192, the model scored 13/30 — on par with standard 8192 decoding.
Links
- Paper: arXiv:2602.16284 — https://arxiv.org/abs/2602.16284
- Code repository: https://github.com/adamzweiger/compaction
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177168716