The Problem: Context Overflow in Long-Horizon Coding Agents
When an AI coding agent optimizes a CUDA kernel, it reads code, runs tests, inspects errors, and edits code — each step inflating the context window. By 200 steps, the model's maximum context is blown: on KernelBench Level 3, 98% of uncompressed runs terminate early from context overflow. A baseline agent manages only a median of 99 steps and a 1.30× kernel speedup.
The core tension: agents need memory, memory grows until it must be compressed, and compression loses information.
The 'Summary of Summaries' Problem
The intuitive fix — LLM summarization of old context (as in Claude Code's auto-compaction) — suffers from a telephone-game effect. Each round of summarization loses details: variable names vanish, call relationships blur, error messages disappear. Worse, the model *believes* it has complete information and won't re-check sources. The paper calls this low compaction precision: it's not how much you lose, but that you don't know what you lost.
CliffCompaction, from Trang Nguyen and Tim Dettmers (of QLoRA fame) at CMU and Bosch Center for AI, proposes a counterintuitive rule: never compress compressed content.
How It Works: Truncate, Don't Rewrite
1. Trigger compaction when context hits a threshold (e.g., 128K tokens) 2. Drop tool results longer than 500 characters; reduce tool calls to signatures (name + args); truncate agent thinking to 300 characters 3. Keep the system prompt and task description intact; keep the last K turns verbatim 4. The key rule: at each compaction, discard the previous compaction block entirely and only compress the 'live session' since the last compaction
Rule 4 is the soul of the method. No information is explicitly carried forward, but each round's agent behavior was shaped by the previous block, and the new block is generated from that behavior. The paper calls this 'residual propagation':
Precision vs. Recall
The paper frames compaction as a precision/recall tradeoff:
- Compaction precision: is the retained information faithful? Summarization has high recall but low precision — distortion compounds across nested summaries.
- Compaction recall: how much history remains retrievable after multiple compactions? CliffCompaction has high precision (verbatim truncation, no rewriting) but low recall — after two compactions, early information is gone entirely.
- Kimi K2.6 + CliffCompaction (16K threshold): 61.42% pass rate, beating the full-context baseline (59.16%) while cost drops from $0.40 to $0.19
- On Claude Code with GLM 5.3 Flash: 76.69% at ~45K average peak context, beating native auto-compaction (70.97%) and default 200K config (73.03%)
- Uncompressed Kimi K2.6 × 3 rollouts: $91.65 for +4.8 points; a single GPT 5.3 Codex run is cheaper and better — naive scaling doesn't pay
- Compressed Kimi K2.6 × 3 rollouts: $58.01 for +10.5 points — cheaper than one Codex run and matching Opus 4.7 (69.7% vs 69.4%)
- A Soft Group Verification (SGV) selector (LightGBM on rollout features: step count, tool calls, code-line overlap) picks the best rollout. Cross-model transfer works — selectors trained on GLM rollouts work for Kimi — suggesting the signal is task-intrinsic.
- Single trajectories exceed 1 million tokens (15–70× the SWE-bench median)
- Uncompressed Kimi K2.7 (256K window): 99 steps, 1.30× speedup. CliffCompaction (128K threshold): 200 steps → 2.23×; 400 steps → 3.58×, beating AdaExplore (1.78×) and CUDA-Agent (1.80×)
- Coverage: at 400 steps, 92% of kernels exceed 1.2× and 86% exceed 2× (vs. 36% and 22% for AdaExplore)
For coding agents, precision beats recall because tasks have execution feedback: forgotten file contents can be re-read, but *wrong* remembered information drives wrong decisions silently.
KV-Cache Economics
The 'cliff' name comes from the context-length curve: growth, sudden drop, growth, drop. This shape is deliberate — between compactions the context is immutable, so the KV-cache stays valid. Summarization rewrites invalidate the entire cache, forcing expensive full re-prefills (uncached input tokens cost 5–6× more than cached ones). CliffCompaction minimizes cache invalidations and makes each one cheap, since the post-compaction context is small.
Results
Terminal-Bench 2.0
Test-Time Scaling Economics
KernelBench Level 3
Engineering Insights
Where tokens go: on Terminal-Bench, tool results are 56% and tool calls 28% of tokens (84% combined) — exactly what CliffCompaction targets.
Why not compress thinking? Thinking is the agent's own reasoning chain (irrecoverable); tool results are external and re-fetchable. Rule: compress aggressively what's recoverable; preserve what's not.
Scaffold-agnostic: implemented as an API proxy, it works even with closed-source agents like Claude Code — no schema knowledge needed, a true drop-in solution.
8K is a stress test, not a working point: performance degrades gradually; 16K–32K is the practical sweet spot.
Closing Thoughts
The deeper question: what should be stored explicitly vs. propagated implicitly? CliffCompaction's answer — recent high-fidelity context explicit, distant context implicit through behavioral residue — mirrors human memory.
It's also a win for simplicity: no trained models, no auxiliary LLM calls, no external memory store — just a few rules — yet it beats trained search algorithms (AdaExplore), summarization pipelines, and RAG-based memory systems. Occam's razor applied to agent engineering: ask what the agent actually needs to remember.
Most consequentially, CliffCompaction turns test-time scaling from an academic toy into an engineering reality, cutting 3× rollout cost to 1.9×. If the trend holds, the next scaling dimension may be more rollouts of smaller models, not bigger models.
---
Paper: CliffCompaction: Cost-Efficient Compaction for Long-Horizon Coding Agents Authors: Trang Nguyen, Eulrang Cho, Bingqing Chen, Tim Dettmers (CMU + Bosch Center for AI) arXiv: 2609.26779 Code: github.com/nguyenvuthientrang/cliffcompaction