AutoCompact: Teaching AI Coding Assistants When to Forget

Have you ever opened dozens of browser tabs, each one "might be useful later," until your browser slows to a crawl and you have to close some—but you don't know which ones? What you hesitate over isn't "whether to close" but "when to close, and which ones."

AI coding assistants face the same problem. When helping you fix a complex bug, they need to read code, run tests, inspect errors, read more code... every step stuffs more content into the context window. Once it's full, the model can't "see" everything anymore and performance degrades. The obvious fix is to compress the context, turning the verbose history into a short summary. But the questions arise: when to compress? What to compress? And how to continue afterward?

Existing approaches are either "compress when length hits a threshold" (length-triggered) or "let the model decide on its own" (proactive). The former is too mechanical; the latter too arbitrary. A paper from an Alibaba team proposes AutoCompact, which lets the model learn *when to compress* via reinforcement learning—and, crucially, the compaction behavior and the coding behavior are jointly trained under the same task reward, with no extra reward engineering.

Core Mechanism: One compact() Action, Three Decisions in One

AutoCompact's design is so simple it barely looks like a new method. It adds one special action to the model's action space: compact(). When the model invokes it, the current context is replaced with a model-generated summary, and the model continues working from that summary.

That simple? No. The key is how it's trained.

Traditional approaches design a separate reward for compression: is the summary good? Was information lost? AutoCompact does none of that. It uses GRPO (Group Relative Policy Optimization), with the sole reward signal being whether the final task passes tests. Fix the bug, get 1 point; fail, get 0.

This forces the model to learn three things simultaneously: 1. When to compress — compressing too early loses information; too late overflows the window 2. What to compress — which information matters for the task ahead, and which can be dropped 3. How to continue afterward — resume work from the summary instead of "forgetting what it was doing"

All three share the same reward signal. It's like teaching someone to cook by only telling them whether the final dish tastes good—never saying "chop evenly" or "watch the heat." They have to figure out the relationship between chopping and heat themselves.

Training Details: Segmenting Trajectories, Sharing Advantages

One technical detail deserves unpacking. After the model calls compact(), the context is rewritten—the prior conversation becomes a summary. The trajectory is no longer a straight line; it's "cut" at each compaction point.

AutoCompact's approach: split the trajectory into segments at each compaction. Within each segment, the prefix the model sees only grows (normal autoregressive generation). Across segments, the same trajectory-level advantage is shared.

In other words: the compaction decision, the summary content, and the post-compaction behavior all receive the same "did the task succeed" signal. If the task ultimately succeeds, good compaction decisions are reinforced; if it fails, bad ones are suppressed. The model never needs to be told "you compressed well or badly"—it learns purely from the final outcome.

This design avoids the "reward engineering" trap—no need to build a "summary quality evaluator," no worry about conflicts between compression reward and task reward. The task reward is the only teacher.

Results: A 9.2% Absolute Gain

On SWE-bench Verified, AutoCompact achieves a 39.6% pass rate, up from the base model's 30.4%—a 9.2 percentage-point improvement. On SWE-PolyBench Verified, the pass rate rises from 19.5% to 24.5%, a 5.0-point gain.

These numbers alone aren't startling. What's really interesting is the ablation:

Fixed Compaction actually hurts performance—1.6% and 0.9% *below* the base model on the two benchmarks respectively. This shows that "blind compression" is harmful: compress when you shouldn't, and you lose critical information.

CompactionRL (RL training but with fixed compression triggers) does better than Fixed (+3.9% and +1.2%), but still far behind AutoCompact. RL alone isn't enough—the key is letting the model decide for itself when to compress.

SFT builds the base behavior; RL refines it. AutoCompact-SFT (supervised training only) already uses compaction on 44.3% of tasks; RL pushes this to 58.5%. More importantly, RL reduces information omissions in summaries: critical state omissions drop from 3.1% to 0.2%, and next-action omissions from 8.2% to 2.2%.

Cost Efficiency: Compression Isn't Just About Performance

Beyond pass rates, AutoCompact shows significant cost-efficiency gains. At a $0.10 inference budget, AutoCompact holds a 19.9% advantage over the base model; even at a $4.00 budget, it retains a 1.9% advantage.

This means compression isn't merely an "emergency measure when the context is full"—it's an active cost-optimization strategy. Compressing redundant information early in the task saves substantial token cost without sacrificing performance.

Why This Matters

The paper's core contribution isn't "proposing context compression"—that's been done. It's a design philosophy: treat context management as part of the task to be trained, not a separate subproblem to be solved.

The traditional view: compression is compression, the task is the task, each managed separately. AutoCompact says: no—compression decisions are themselves part of task execution. A good engineer doesn't just fix bugs; they know when to clear the workbench, when to reread the code. This kind of "metacognitive" ability shouldn't be trained in isolation—it should emerge naturally within the task.

This points to a larger pattern: "forgetting" in AI systems isn't a bug, it's a feature. Human brains forget constantly—not because we can't remember, but because remembering everything would make retrieval impossible. The context window's size limit, in a sense, "forces" the model to learn to forget—which may actually be a good thing.

Admittedly, AutoCompact has limitations. It's validated only on coding tasks—does it transfer to dialogue, reasoning, or creative domains? How to evaluate summary quality (the paper provides no human evaluation)? Is the RL training compute affordable? These questions are left to future work.

But it proves at least one thing: letting the model learn for itself "when to forget" is far better than telling it "compress at N tokens".

---

Paper: https://arxiv.org/abs/2610.02163

Project page: https://autocompact.github.io