Self-GC Deep Dive: Applying Java GC Ideas to LLM Agent Context Management
> Speaker: Hao Xubin, AI Engineering Architect at Xiaohongshu > Venue: AiCon Global AI Development and Application Conference (June 26-27, 2026, Shanghai) > One-line positioning: Borrowing Java GC concepts to manage multi-turn Agent session context as objects, combined with prefix-cache constraints, achieving a 15%-20% net input TPM reduction
1. The Nature of the Problem: "Memory Leaks" in Long-Running Agents
Key figures cited from the talk's material:
- Average input: ~70k tokens
- Average TPM: ~100 million tokens
- The system bottleneck is shifting from "per-step model capability" to "whether the system can run stably long-term under limited context, cache windows, and continuous tool interactions"
- New objects are created and destroyed quickly in the Eden space
- Surviving objects are promoted to Survivor spaces
- Long-lived objects enter the Old Generation
- GC is not a one-time event but a generational, incremental, concurrent continuous process
- prune: delete low-value segments (e.g., duplicate tool-call results, already-confirmed intermediate steps)
- mask: keep segments but mark them as "read-only summaries", without expanding full content
- fold: collapse multiple related segments into a structured summary
- plan phase: evaluate which segments can be compressed, how, and with what expected information loss
- commit phase: actually execute compression, but only after verifying compression quality meets the bar
- Monitor prefix-cache hit patterns
- Within cache-hit "hot zones", avoid any compression that would change the prefix
- Execute backlogged compression tasks only when the cache naturally invalidates (TTL expiry, new session start)
- Average input: 70k tokens
- Average TPM: 100M tokens
- Assumed 100:1 input/output ratio
- Each turn sends 70k input + 700 output
- After 10 turns, input bloats to 700k (full history stacking)
- Assuming API pricing of $0.01/1k input tokens
- 10-turn cost = 700k × $0.01/1k = $7.0
- Through progressive compression, effective input after 10 turns is held at 560k-595k (15%-20% reduction)
- 10-turn cost = 560k × $0.01/1k = $5.6
- Savings of $1.4 per session
- 15%-20% fewer input tokens per minute = 1.5M-2.0M tokens/minute saved
- At $0.01/1k = $15-$20 saved per minute
- Annualized savings of roughly $788k-$1.05M (assuming 24×7 operation)
- Context stacks without limit; error or truncation when the window fills
- Example: early ChatGPT API usage
- Threshold-triggered one-shot compression (Claude Code, Gemini CLI, OpenClaw /compact)
- Problem: uncontrolled timing, unevaluated information loss
- Continuous monitoring, progressive compression, cache coordination
- Introduces the "object lifecycle" concept, akin to Java GC's generational management
- Virtual memory: contexts beyond the window automatically paged out to external storage (vector DB, graph DB, filesystem)
- Paged management: context divided into fixed-size "pages", loaded on demand
- Reference counting + garbage collection: automatic identification and reclamation of unreferenced context
- Cache hierarchy: L1 (current window), L2 (recent cache), L3 (external storage)
- Process isolation: context isolation and sharing mechanisms between Agents/tools
- Snapshots and rollback: context snapshots at key decision points for backtracking
- Talk: AiCon 2026, Hao Xubin, "Self-GC: A Multi-Turn Agent Context Governance Scheme Combined with Prefix Cache Constraints"
- Technical background: Java GC (generational, incremental, concurrent), vLLM Prefix Caching (Radix Tree), Claude Code's 8-part summary, Gemini CLI's 70/30 strategy
- Industry practice: Manus KV-cache hit-rate optimization, OpenClaw context compaction, Mem0 cross-session memory
This is not just performance tuning—it is a survival problem.
When an Agent runs 10, 20, or 100 turns, context snowballs. Every turn must "look back" at all prior actions and observations, with input-to-output ratios reaching 100:1. That means:
> For every 1 output token generated, 100 context tokens must be processed.
The conventional solution is "context compression"—a one-shot compaction once the window fills. The Self-GC team identified an overlooked gap:
Existing work mostly focuses on final compaction near the context limit, but insufficient attention has been paid to the pre-compaction tidying layer and to coordinating the compression process with prompt caches.
This is like a Java program only triggering Full GC when memory is exhausted, rather than doing continuous Minor GC. Self-GC's insight: context governance should happen during runtime, not as firefighting at the moment of crisis.
2. Self-GC's Core Design: From "Garbage Collection" to "Context Collection"
Intellectual Roots: Java GC's Object Lifecycle Management
Java GC's essence is not "deleting garbage" but continuous lifecycle management of runtime objects:
Self-GC maps this onto Agent context:
| Java GC Concept | Self-GC Mapping | |---|---| | Object | Context segment (dialogue turn, tool call, observation) | | Reference counting | Explicit addressing (which segments are referenced by later turns) | | Fast Eden reclamation | Low-loss prune/mask/fold (lightweight compression) | | Survivor promotion | Important context enters a "retention layer" | | Old Generation | Long-term memory / external storage (beyond the context window) | | Full GC | Final compaction (heavy compression near the window limit) | | Concurrent marking | plan/commit decoupling (mark first, execute later) |
Four Core Mechanisms
1. Explicit Addressing
Traditional context management is "linear stacking"—every turn is appended to the end. Self-GC gives each context segment a unique "address", so later turns can reference history via pointers instead of copying it.
This is a shift from "pass-by-value" to "pass-by-reference": no need to copy entire history into the current prompt—just a pointer.
2. Low-loss prune / mask / fold (three-stage lightweight compression)
The key is "low-loss"—not blunt truncation, but selectively preserving information density.
3. plan/commit decoupling
Borrowing two-phase commit from database transactions:
This gives the system a chance to "back out"—if planning finds a segment is frequently referenced, its compression is postponed.
4. Cache-aware delayed commit
Self-GC's most elegant design: compression operations are not executed immediately, but deferred into the "gaps" when the prefix cache is invalidated anyway.
Prefix caching works by reusing the KV cache when a request's prefix matches a cached one, skipping redundant computation. But modifying earlier context mid-conversation changes the prefix and invalidates the cache.
Self-GC's cache-aware delayed commit strategy:
This resembles Java's concurrent mark-sweep: run GC during off-peak periods to avoid impacting online latency.
3. Why Not "Compress Only When Full"?
Problems with traditional compaction
| Problem | Description | |---|---| | Latency spikes | Compression is compute-intensive; triggering near the window limit causes latency to soar | | Concentrated information loss | Compressing too much at once easily loses critical information | | Cache thrashing | Large-scale context edits frequently invalidate the prefix cache | | Indiscriminate compression | No distinction between important and unimportant content; coarse strategy |
Self-GC's progressive advantages
| Dimension | Traditional compaction | Self-GC | |---|---|---| | Timing | Passive (window full) | Proactive (continuous during runtime) | | Granularity | Coarse (whole-segment truncation/summary) | Fine (segment-level prune/mask/fold) | | Cache friendliness | Poor (large prefix edits) | Excellent (cache-aware delayed commit) | | Information retention | Low (indiscriminate) | High (explicit addressing + plan/commit evaluation) | | Latency impact | High (concentrated computation) | Low (incremental, deferred execution) |
4. Industry Landscape: Where Self-GC Sits
Comparison with existing solutions
| Solution | Strategy | Difference vs. Self-GC | |---|---|---| | Claude Code | 92% threshold + 8-part structured summary | Fixed compression threshold, no cache awareness | | Gemini CLI | 70% threshold + 5-part summary + file-system persistence | Triggers earlier, but still passive | | Manus | Context state machine + masking instead of deletion | Focuses on tool management, not general context governance | | OpenClaw /compact | User/system-triggered compression | Manual/semi-automatic, no runtime governance | | Mem0 / MemOS | Cross-session memory storage + on-demand recall | Solves cross-session memory, not within-session bloat | | LLMLingua | Prompt compression (token-level) | Pure compression algorithm, no Agent runtime awareness |
Self-GC's unique positioning
Self-GC is not "another compression algorithm"—it is arguably the first scheme to systematically introduce runtime object lifecycle management into Agent context governance. Its distinctive traits:
1. Continuous runtime governance: daily maintenance, not crisis firefighting 2. Cache-cooperative design: compression is deeply coordinated with the prefix cache 3. Model-agnostic: a harness-level capability, not bound to a specific model 4. Reversible and evaluable: plan/commit decoupling provides both "undo" and "effect evaluation" mechanisms
5. Quantified Gains: What Does a 15%-20% Net Input TPM Reduction Mean?
Scenario math (based on the cited figures)
Traditional cost structure:
After Self-GC optimization:
At scale (100M TPM ≈ 1428 turns/minute at 70k input each):
This excludes additional savings from prefix-cache hits (cache reads typically cost ~10% of normal).
6. A Deeper Question: What Is the Endgame of Context Governance?
Self-GC reveals a paradigm evolution from "memory management" toward "operating-system-ization" of Agent context:
Stage 1: No management (primitive)
Stage 2: Passive compression (current mainstream)
Stage 3: Runtime governance (where Self-GC sits)
Stage 4: Context operating system (the future)
The author's speculated endgame—a full "Context OS" featuring:
Self-GC represents Stage 3. It demonstrates: context governance is not a "compression algorithm" problem—it is a "runtime system" problem.
7. Feynman-Style Takeaway
> Self-GC's essence: turning "toss everything when it's full" into "tidy as you go." > > Your room doesn't become a garbage heap because you skip one day of cleaning—it's because you do minor GC every day. Agent context is the same: don't panic-compress at 200k tokens; tidy up the unimportant stuff after every turn. > > Even better, Self-GC recognizes the tension between "tidying" and "caching": you want to organize the room, but you worry that putting frequently used items away means you can't find them next time. So the solution is—do the heavy reorganizing only "when you're out the door" (during cache invalidation gaps), and in normal times only pick up the obvious trash. > > This isn't just technical innovation—it's engineering intuition.