Self-GC Deep Dive: Applying Java GC Ideas to LLM Agent Context Management

> Speaker: Hao Xubin, AI Engineering Architect at Xiaohongshu > Venue: AiCon Global AI Development and Application Conference (June 26-27, 2026, Shanghai) > One-line positioning: Borrowing Java GC concepts to treat multi-turn Agent session context as managed objects, combined with prefix-cache constraints to achieve a 15%-20% net input TPM reduction

Key points

• Long-running agents exhibit a "memory leak" problem: with average input around 70k tokens and ~100M tokens per minute, input-to-output ratios can reach 100:1 — generating 1 output token requires processing 100 context tokens.
• Existing solutions (Claude Code, Gemini CLI, Manus, OpenClaw /compact, Mem0, LLMLingua) mostly perform final compaction when the context nears its limit; the pre-compaction governance layer and coordination with prompt caching are under-explored.
• Self-GC maps Java GC concepts onto context management:

| Java GC concept | Self-GC mapping | |---|---| | Objects | Context segments (turns, tool calls, observations) | | Reference counting | Explicit addressing (which segments later turns reference) | | Fast Eden collection | Low-loss prune/mask/fold (lightweight compression) | | Survivor promotion | Important context moves to a retention layer | | Old Generation | Long-term memory / external storage beyond the window | | Full GC | Final compaction near the window limit | | Concurrent marking | plan/commit decoupling (mark first, execute later) |

The four core mechanisms

1. Explicit addressing — each context segment gets a unique address so later turns reference history by pointer instead of copying it into the prompt (like switching from pass-by-value to pass-by-reference). 2. Low-loss prune/mask/fold — *prune* deletes low-value segments (duplicate tool results, confirmed intermediate steps); *mask* keeps segments but marks them as read-only summaries; *fold* merges related segments into structured summaries. The key is selective information preservation, not brute truncation. 3. plan/commit decoupling — inspired by two-phase transaction commits: the plan phase evaluates what to compress and expected information loss; the commit phase executes only after quality verification. Frequently referenced segments can be deferred. 4. Cache-aware delayed commit — prefix caches only hit when the request prefix is unchanged; modifying context invalidates them. Self-GC monitors cache hit patterns, avoids prefix-changing compression in hot zones, and executes backlog compression only when the cache naturally expires (TTL, new session) — analogous to concurrent mark-sweep during low-traffic windows.

Why not "compress when full"?

| Dimension | Traditional compaction | Self-GC | |---|---|---| | Timing | Passive trigger (window full) | Continuous runtime governance | | Granularity | Coarse (truncate/summarize whole blocks) | Fine (segment-level prune/mask/fold) | | Cache friendliness | Poor (large prefix changes) | Good (cache-aware delayed commit) | | Information retention | Low (indiscriminate) | High (explicit addressing + plan/commit evaluation) | | Latency impact | High (concentrated computation spikes) | Low (incremental, deferred) |

Industry positioning

• Claude Code: triggers at 92% threshold with 8-section structured summaries — fixed threshold, no cache awareness.
• Gemini CLI: 70% threshold + 5-section summaries + filesystem persistence — earlier trigger but still passive.
• Manus: context state machine + masking instead of deletion — focused on tool management.
• OpenClaw /compact: manual/semi-automatic compaction, no runtime governance.
• Mem0 / MemOS: cross-session memory, not intra-session bloat.
• LLMLingua: token-level prompt compression, no agent runtime awareness.

Self-GC's distinct claims: runtime continuous governance, deep coordination with prefix caching (e.g., vLLM Radix Tree prefix caching), model-agnostic harness-level design, and reversibility/evaluability via plan/commit.

Quantified benefits

With ~70k average input tokens and a hypothetical $0.01/1k input pricing: a 10-turn session with fully stacked history (~700k input tokens) costs about $7.0; a 15%-20% reduction brings it to ~$5.6-$5.95. At 100M TPM, that is roughly 1.5M-2.0M input tokens saved per minute (~$15-$20/minute, an estimated ~$788k-$1.05M annualized at 24×7 operation) — before accounting for additional savings from prefix cache hits (cache reads typically cost ~10% of normal).

The bigger picture: toward a Context OS

The author frames context management as a paradigm evolution:

1. Unmanaged — unlimited stacking, truncation on overflow (early ChatGPT API usage). 2. Passive compression — threshold-triggered one-shot compaction (current mainstream). 3. Runtime governance — continuous, incremental, cache-coordinated management (Self-GC's stage). 4. Context operating system (speculative future) — virtual memory with spillover to vector/graph DBs, page-based management, reference counting + GC, cache hierarchies (L1 window / L2 recent cache / L3 external storage), process isolation between agents, snapshots and rollback.

Feynman-style takeaway

> Self-GC's essence: turning "throw everything out when full" into "tidy as you go." Your room doesn't become a garbage dump because you do minor GC daily. And because tidying conflicts with the cache (putting frequently used things away makes them harder to find), Self-GC only does major reorganization during natural cache-invalidation windows —平时 just clearing obvious trash.

This is less a novel algorithm than a piece of engineering intuition: context governance is a runtime systems problem, not just a compression problem.

References

• Talk: AiCon 2026, Hao Xubin, "Self-GC: A Multi-turn Agent Context Governance Approach Incorporating Prefix Cache Constraints"
• Background: Java GC (generational, incremental, concurrent), vLLM Prefix Caching (Radix Tree), Claude Code 8-section summaries, Gemini CLI 70/30 strategy
• Industry practice: Manus KV-cache hit-rate optimization, OpenClaw context compaction, Mem0 cross-session memory