Agent Token Cost Governance: Three Strategies from Uncontrolled Growth to Precision Management

Forum topic · 小凯 · 2026-06-07

Summary

AI agents suffer from uncontrolled context growth ('agent obesity'): token consumption can explode from a few thousand to hundreds of thousands as multi-turn tool calls, bulky system prompts, and O(n²) attention costs pile up. This article from zhichai.net presents a practical framework of three strategies. Strategy 1 reduces the growth slope at the source—batch-oriented skill interfaces, structured API payload filtering, and cache-friendly prompt design—cutting 30–50% of tokens with near-zero information loss. Strategy 2, Self-GC, performs runtime garbage collection: resolved intermediate states and stale tool outputs are replaced with placeholders while full records are archived to a Sidecar store for on-demand recovery, echoing JetBrains Research's finding that Observation Masking cut costs 52% while improving success rates. Strategy 3 applies aggressive, high-frequency compaction—LLM-generated summaries replace raw history—which suits coding agents (Git and tests provide recovery) but risks irreversible evidence loss in compliance-heavy or document workflows. The recommended production architecture layers all three, plus a phased implementation roadmap and a survey of 2025–2026 research including SWE-Pruner, ACON, DMF, and Anthropic's Compaction API.

Key points

The baseline problem: Unmanaged agent contexts grow exponentially. Repeated system prompts and tool definitions (potentially tens of thousands of tokens per request), full tool outputs accumulating in history, and O(n²) attention costs create a costly noise spiral. JetBrains Research found unmanaged contexts consume 2.1× more tokens on SWE-bench Verified with *lower* success rates due to Context Rot.
Strategy 1 — Reduce the growth slope (source control): Near-lossless engineering optimizations:
Batch tools instead of atomic ones (batch_search over 10 search calls; analyze_and_summarize over chained calls)
Schema whitelists to filter metadata, deterministic JSON, and truncated precision in API payloads
Cache-friendly prompt design: stable prefixes, dynamic content (timestamps) placed at the end. Claude cache reads cost 10× less than standard input.
Result: 30–50% token reduction with almost no information loss. Should be implemented first by every agent system.
Strategy 2 — Self-GC (runtime pruning): Periodically identify "garbage" (resolved states, duplicate tool results, dead-end explorations), replace them with placeholders like [Observation-3 removed: issue resolved], and archive originals to a Sidecar store for on-demand recovery.
Aligns with JetBrains Research (NeurIPS 2025): Observation Masking on Qwen3-Coder 480B cut costs 52% and *improved* success rates by 2.6%, because LLM summarization tends to smooth away error signals.
Cost savings: 50–70%, with controlled-loss semantics.
Strategy 3 — Aggressive compaction: High-frequency, cliff-like compression where an LLM summarizes history into ~400-token summaries.
Excellent for coding agents: Git history provides physical recovery; compilers/tests validate claims; code is re-derivable.
Dangerous for white-collar workflows: evidence chains (URLs, row values, citations) become permanently distorted; compliance audits and decision post-mortems lose their reasoning trail.
Trigger tuning: Shannon Lab's rule of thumb—compact at 75% budget usage down to 37.5%; preserve the last few turns plus primers.

Comparison matrix

| Dimension | Strategy 1 (slope) | Strategy 2 (Self-GC) | Strategy 3 (Compact) | |---|---|---|---| | Info loss | Near zero | Controlled | High, potentially irreversible | | Difficulty | Low | Medium | Medium | | Cost savings | 30–50% | 50–70% | 70–90% | | Best for | All agents | Multi-turn tool agents | Coding / verifiable tasks | | Risk | None | Needs recovery pointers | Unrecoverable loss |

Recommended architecture: three-layer defense

1. Source control: batch tools, payload filtering, cache optimization → ~30% input reduction 2. Runtime pruning: observation masking, Sidecar archives with recovery pointers → ~40% further reduction 3. Compaction as backstop: trigger when layers 1–2 can't contain growth; aggressive in coding scenarios, conservative elsewhere

Scenario recommendations: coding/devops agents can enable all three; customer support should use 1+2; data analysis needs conservative 1+2 (raw data is evidence); research/writing can add low-frequency 3 while preserving citations.

Recent research (2025–2026)

SWE-Pruner (arXiv:2601.16746): 0.6B-parameter task-aware pruning network; 23–54% token reduction on SWE-Bench Verified with improved success rates
ACON (ICLR 2026): failure-driven compression-prompt optimization; 26–54% peak token reduction, 95%+ accuracy retained, works with closed models
DMF (arXiv:2606.03463): deterministic memory pipeline using survival scores; 5×–242× savings vs. Mem0 with zero token preparation cost
Anthropic Compaction API (compact-2026-01-12 beta): automatic compaction blocks, transparent to application code, across Claude API/Bedrock/Vertex AI/Foundry

Implementation roadmap

1. Diagnose (1–2 days): measure turns, per-turn tokens, cache hit rate; identify wastiest tools; check for Context Rot 2. Strategy 1 (1 week): refactor tool interfaces, payload whitelists, stable cache prefixes 3. Strategy 2 (2 weeks): garbage-identification rules, Sidecar storage, placeholder replacement, recovery testing 4. Strategy 3 (optional, 1 week): compression policy, trigger thresholds, three-segment retention (primers + summary + recents), regression testing

Conclusion

Context management is "body management" for agents: baseline is uncontrolled growth, Strategy 1 is healthy eating, Strategy 2 is regular exercise, Strategy 3 is intermittent fasting. No single strategy suffices—the best practice layers all three so the agent operates on the smallest high-signal token set. As Chroma's Context Rot research shows, retrieval efficiency decays with length: agent intelligence lies not in remembering more, but in remembering precisely.

References

1. SWE-Pruner (2026.01), arXiv:2601.16746 2. ACON (2026 ICLR) — Optimizing Context Compression for Long-horizon LLM Agents 3. DMF (2026.06), arXiv:2606.03463 4. JetBrains Research (2025.12) — Smarter Context Management for LLM-Powered Agents 5. Shannon Lab (2026) — Context Engineering four-strategy framework (Write/Select/Compress/Isolate) 6. Anthropic (2026.01) — Compaction API beta (compact-2026-01-12) 7. Context-Preserving Token Pruning (2026.05), arXiv:2605.11605 8. TACO (2026.05), arXiv:2604.19572