Self-GC Deep Dive: Applying Java GC Ideas to LLM Context Management
> Speaker: Hao Xubin, AI Engineering Architect at Xiaohongshu > Venue: AiCon Global AI Developer Conference (June 26-27, 2026, Shanghai) > One-line positioning: Borrowing Java GC concepts to govern multi-turn Agent session context as objects, combined with prefix cache constraints, achieving a 15%-20% net reduction in input TPM
1. The Essence of the Problem: Long-Running Agents Suffer from "Memory Leaks"
Key data points from the talk:
- Average input: ~70k tokens
- Average TPM: ~100 million tokens
- The system bottleneck is shifting from "single-step model capability" to "whether the system can run stably long-term under limited context, cache windows, and continuous tool interactions"
- Newly created objects are quickly created and destroyed in the Eden space
- Surviving objects are promoted to Survivor spaces
- Long-lived objects enter the Old Generation
- GC is not a one-off event but a generational, incremental, concurrent, continuous process
- prune: remove low-value fragments (e.g., duplicate tool results, already-confirmed intermediate steps)
- mask: keep the fragment but mark it as a "read-only summary," without expanding full content
- fold: collapse multiple related fragments into a structured summary
- plan phase: evaluate which fragments can be compressed, how, and the expected information loss
- commit phase: actually execute compression, but only after verifying compression quality is acceptable
- Monitor prefix cache hit patterns
- Within cache "hot zones," avoid any compression that would alter the prefix
- Execute backlogged compression tasks only when the cache naturally invalidates (TTL expiry, new session start)
- Average input: 70k tokens
- Average TPM: 100M tokens
- Assume a 100:1 input/output ratio
- Each turn sends 70k input + 700 output
- After 10 turns, input accumulates to 700k (full history stacking)
- 10-turn cost = 700k × $0.01/1k = $7.0
- Progressive compression keeps effective input at 560k-595k over 10 turns (15%-20% reduction)
- 10-turn cost = 560k × $0.01/1k = $5.6
- Savings: ~$1.4 per session
- 15%-20% input reduction = 1.5M-2.0M tokens saved per minute
- At $0.01/1k = $15-$20 saved per minute
- Annualized savings of roughly $788k-$1.05M (assuming 24×7 operation)
- Stage 1 — Unmanaged (primitive): context stacks infinitely; errors or truncation when full
- Stage 2 — Passive compression (current mainstream): threshold-triggered one-shot compression (Claude Code, Gemini CLI, OpenClaw /compact); uncontrolled timing, unevaluated loss
- Stage 3 — Runtime governance (where Self-GC sits): continuous monitoring, progressive compression, cache coordination; generational lifecycle management like Java GC
- Stage 4 — Context Operating System (future): a full "Context OS" with virtual memory (spilling to vector DBs/graph DBs/filesystems), paged management, reference counting + garbage collection, cache hierarchy (L1 window / L2 recent cache / L3 external storage), process isolation between Agents/tools, and snapshot-and-rollback
- Talk: AiCon 2026, Hao Xubin, "Self-GC: A Multi-Turn Agent Context Governance Scheme Incorporating Prefix Cache Constraints"
- Technical background: Java GC (generational, incremental, concurrent), vLLM Prefix Caching (Radix Tree), Claude Code 8-section summarization, Gemini CLI 70/30 strategy
- Industry practice: Manus KV cache hit-rate optimization, OpenClaw context compaction, Mem0 cross-session memory
This is not performance tuning — it is a survival problem.
After an Agent runs 10, 20, or 100 turns, context snowballs. Every turn must "look back" at all previous actions and observations, with input-to-output ratios reaching 100:1. That means:
> For every 1 token generated, ~100 tokens of context must be processed.
The traditional solution is "context compression" — compress everything at once once the window is full. But the Self-GC team identified a neglected gap:
Most existing work focuses on final compaction once context approaches the limit, but insufficient attention has been paid to the pre-compaction tidying layer, and to coordinating the compression process with the Prompt Cache.
It is like a Java program that only triggers Full GC when memory runs out, instead of continuously performing Minor GC during normal operation. Self-GC's insight: context governance should be moved up into the running process, not saved for the crisis moment.
2. Self-GC's Core Design: From "Garbage Collection" to "Context Collection"
Intellectual Origin: Java GC's Object Lifecycle Management
Java GC's core is not "deleting garbage" but continuous lifecycle management of runtime objects:
Self-GC maps this to Agent context:
| Java GC Concept | Self-GC Mapping | |---|---| | Object | Context fragment (dialogue turn, tool call, observation) | | Reference counting | Explicit addressing (which fragments are referenced by later turns) | | Fast Eden reclamation | Low-loss prune/mask/fold (lightweight compression) | | Survivor promotion | Important context enters a "retention layer" | | Old Generation | Long-term memory / external storage (beyond the context window) | | Full GC | Final compaction (heavy compression when the window nears its limit) | | Concurrent marking | plan/commit decoupling (mark first, execute later) |
Four Technical Mechanisms
1. Explicit Addressing
Traditional context management "linearly stacks" — each turn is appended to the end. Self-GC introduces explicit addressing, giving each context fragment a unique "address" so later turns can reference history by pointer rather than copy. It is the shift from "pass-by-value" to "pass-by-reference": you don't copy the entire history into the current prompt — you just keep a pointer.
2. Low-Loss prune/mask/fold (Three-Stage Lightweight Compression)
The key is "low-loss" — selective, density-preserving compression rather than brute truncation.
3. plan/commit Decoupling
Borrowed from database two-phase commit:
This gives the system a chance to "regret" — if the plan finds a fragment frequently referenced, compression on it is postponed.
4. Cache-Aware Delayed Commit
The most elegant design: compression is not executed immediately, but delayed to the natural "gaps" when the prefix cache invalidates.
A prefix cache works by reusing the KV cache whenever a request's prefix matches what's cached. But if earlier context is modified mid-conversation, the prefix changes and the cache is invalidated.
Self-GC's cache-aware delayed commit strategy:
It's like Java's concurrent mark-sweep: run GC during off-peak periods to avoid impacting online latency.
3. Why Not "Compress Only When Full"?
Problems with Traditional Compaction
| Problem | Description | |---|---| | Latency spikes | Compression is compute-intensive; triggering near the window limit causes latency to surge | | Concentrated information loss | Compressing too much at once easily loses key information | | Cache thrashing | Massively modifying context frequently invalidates the prefix cache | | Indiscriminate compression | No distinction between important and unimportant content |
Self-GC's Progressive Advantages
| Dimension | Traditional Compaction | Self-GC | |---|---|---| | Timing | Passive (window full) | Proactive (continuous during runtime) | | Granularity | Coarse (wholesale truncation/summarization) | Fine (fragment-level prune/mask/fold) | | Cache friendliness | Poor (large-scale prefix modification) | Excellent (cache-aware delayed commit) | | Information retention | Low (indiscriminate) | High (explicit addressing + plan/commit evaluation) | | Latency impact | High (concentrated computation) | Low (incremental, deferred execution) |
4. Industry Landscape: Where Self-GC Sits
| Approach | Strategy | Difference from Self-GC | |---|---|---| | Claude Code | 92% threshold trigger + 8-section structured summary | Fixed threshold timing, no cache awareness | | Gemini CLI | 70% threshold + 5-section summary + filesystem persistence | Earlier trigger, but still passive | | Manus | Context state machine + masking instead of deletion | Focuses on tool management, not general context governance | | OpenClaw /compact | User/system-triggered compression | Manual/semi-automatic, no runtime governance | | Mem0 / MemOS | Cross-session memory storage + on-demand recall | Solves cross-session memory, not within-session bloat | | LLMLingua | Prompt compression (token-level) | Pure compression algorithm, no Agent runtime awareness |
Self-GC's Unique Position
Self-GC is not "another compression algorithm" — it is the first to systematically bring runtime object lifecycle management into Agent context governance. Its distinctive traits:
1. Continuous runtime governance: daily maintenance instead of crisis firefighting 2. Cache-co-designed: compression works in deep coordination with the prefix cache 3. Model-agnostic: a harness capability, not tied to any specific model 4. Reversible and evaluable: plan/commit decoupling enables regret and effect evaluation
5. Quantified Benefits: What a 15%-20% Net Input TPM Reduction Means
Based on the talk's figures:
Traditional cost structure (illustrative, at $0.01/1k input tokens):
After Self-GC optimization:
At scale (100M TPM ≈ 1428 turns/min at 70k input per turn):
And this excludes additional savings from prefix cache hits (cached reads typically cost ~10% of normal processing).
6. A Deeper Question: What Is the Ultimate Form of Context Governance?
Self-GC reveals that Agent context management is undergoing a paradigm shift from "memory management" to "operating-system-ization":
Self-GC represents Stage 3. It demonstrates: context governance is not a "compression algorithm" problem — it is a "runtime system" problem.
7. Feynman-Style Summary
> Self-GC's essence: turn "throw out everything when full" into "tidy as you go." > > Your room doesn't become a garbage dump just because you skip one day of cleaning — you do minor GC every day. Agent context is the same: don't panic-compress at 200k tokens; tidy up unimportant things after each turn. > > Even better, Self-GC recognizes the tension between "tidying" and "caching": you want to organize the room, but you worry that putting frequently used things away means you can't find them next time. So the solution is — only do large-scale reorganization "while you're out" (cache invalidation gaps), and just remove the obviously trash in the meantime.