Self-GC Deep Dive: Applying Java GC Ideas to LLM Agent Context Management

> Speaker: Hao Xubin, AI Engineering Architect at Xiaohongshu (RED) > Venue: AiCon Global AI Development & Application Conference (June 26–27, 2026, Shanghai) > One-line positioning: Borrowing Java GC concepts to treat multi-turn Agent session context as managed objects, combined with prefix-cache constraints to achieve a 15%-20% net input TPM reduction

1. The Problem: Long-Running Agents Suffer "Memory Leaks"

Key figures from the talk:

• Average input: ~70k tokens
• Average TPM: ~100 million tokens
• The bottleneck is shifting from "single-step model capability" to "whether the system can run stably long-term under limited context, cache windows, and continuous tool interactions"

This is not performance tuning — it is a survival problem.

As an Agent runs 10, 20, or 100 turns, context snowballs. Every turn must "look back" at all prior actions and observations, with input-to-output ratios reaching 100:1. That means:

> For every 1 token of generated answer, 100 tokens of context must be processed.

The traditional fix is "context compression" — a one-shot compaction when the window fills. But the Self-GC team identified a neglected gap:

Most existing work focuses on final compaction near the context limit, but pays insufficient attention to a pre-compaction grooming layer and to coordination between the compaction process and the Prompt Cache.

It is like a Java program triggering Full GC only when memory is exhausted instead of doing continuous Minor GC. Self-GC's insight: context governance should happen during execution, not as crisis firefighting.

2. Core Design: From "Garbage Collection" to "Context Collection"

Intellectual Roots: Java GC Object Lifecycle Management

Java GC's essence is not "deleting garbage" but continuous lifecycle management of runtime objects: new objects in Eden, survivors promoted, long-lived objects entering Old Generation — a generational, incremental, concurrent, ongoing process.

Self-GC maps this onto Agent context:

| Java GC concept | Self-GC mapping | |---|---| | Objects | Context segments (dialogue turns, tool calls, observations) | | Reference counting | Explicit addressing (which segments are referenced by later turns) | | Fast Eden reclamation | Low-loss prune/mask/fold (lightweight compression) | | Survivor promotion | Important context entering a "retention layer" | | Old Generation | Long-term memory / external storage (beyond context window) | | Full GC | Final compaction (heavy compression near the limit) | | Concurrent marking | plan/commit decoupling (mark first, execute later) |

Four Core Mechanisms

1. Explicit Addressing

Traditional context management is "linear stacking" — every turn is appended. Self-GC gives each context segment a unique "address" so later turns reference history by pointer instead of copying it into the prompt — a shift from "pass-by-value" to "pass-by-reference."

2. Low-loss prune/mask/fold

• prune: delete low-value segments (duplicate tool results, confirmed-successful intermediate steps)
• mask: keep segments but mark them as "read-only summaries" without expanding full content
• fold: collapse multiple related segments into structured summaries

The key is "low-loss": selective preservation of information density, not blunt truncation.

3. plan/commit Decoupling

Borrowing two-phase commit from databases:

• plan: evaluate which segments to compress, how, and expected information loss
• commit: execute compression only after verifying quality meets the bar

This provides a chance to back off — if the plan finds a segment frequently referenced, its compression is deferred.

4. Cache-aware delayed commit

The most elegant piece: compression is not executed immediately but deferred into "gaps" when the prefix cache invalidates.

Prefix caching reuses KV Cache when a request's prefix matches the cached one. Modifying earlier context mid-conversation invalidates the cache. Self-GC's strategy:

• Monitor prefix cache hit patterns
• Avoid any compression that changes the prefix during "hot" hit zones
• Execute backlog compression only on natural invalidation (TTL expiry, new sessions)

Like Java's concurrent mark-sweep: run GC during low-peak windows to protect online latency.

3. Why Not "Compress When Full"?

| Problem with traditional compaction | Explanation | |---|---| | Latency spikes | Compression is compute-intensive; triggering near the limit causes response latency surges | | Concentrated information loss | One-shot compression of too much content risks losing key information | | Cache thrashing | Massive context changes invalidate prefix caches repeatedly | | Indiscriminate compression | No distinction between important and unimportant content |

| Dimension | Traditional compaction | Self-GC | |---|---|---| | Timing | Passive (window full) | Proactive (continuous during runtime) | | Granularity | Coarse (whole-section truncation/summary) | Fine (segment-level prune/mask/fold) | | Cache friendliness | Poor (large prefix changes) | Good (cache-aware delayed commit) | | Information retention | Low (indiscriminate) | High (explicit addressing + plan/commit evaluation) | | Latency impact | High (concentrated compute) | Low (incremental, deferred execution) |

4. Industry Landscape: Where Self-GC Fits

| Approach | Strategy | Difference from Self-GC | |---|---|---| | Claude Code | 92% threshold trigger + 8-part structured summary | Fixed threshold, no cache awareness | | Gemini CLI | 70% threshold + 5-part summary + file-system persistence | Earlier trigger, but still passive | | Manus | Context state machine + masking instead of deletion | Tool-management focused, not general context governance | | OpenClaw /compact | User/system-triggered compression | Manual/semi-automatic, no runtime governance | | Mem0 / MemOS | Cross-session memory storage + on-demand recall | Solves cross-session memory, not within-session bloat | | LLMLingua | Prompt compression (token-level) | Pure compression algorithm, no Agent runtime awareness |

Self-GC's unique positioning:

1. Continuous runtime governance — daily maintenance, not crisis response 2. Cache co-design — compression is deeply coordinated with prefix caching 3. Model-agnostic — a harness capability, not tied to a specific model 4. Reversible and evaluable — plan/commit decoupling enables backtracking and quality assessment

5. Quantified Benefits: What 15%-20% Net Input TPM Reduction Means

Using the reported baseline (~70k avg input, ~100M TPM, 100:1 I/O ratio):

Traditional approach (10 turns):

• Input grows to 700k (full history stacking)
• At $0.01/1k input tokens: $7.0 per session

With Self-GC:

• Effective input after 10 turns: 560k–595k (15%-20% reduction)
• Cost: $5.6 — a $1.4 saving per session

At scale (~100M TPM ≈ 1,428 turns/minute at 70k input):

• 1.5M–2.0M tokens saved per minute
• At $0.01/1k: $15–$20/minute
• Annualized savings of roughly $788k–$1.05M (24×7 operation)

This excludes additional savings from prefix cache hits (cached reads typically cost ~10% of normal pricing).

6. A Deeper Question: The Endgame of Context Governance

Self-GC suggests Agent context management is evolving from "memory management" to "operating system-ification":

1. Unmanaged (primitive): unbounded stacking; errors/truncation when full — early ChatGPT API calls 2. Passive compression (mainstream today): threshold-triggered one-shot compaction — Claude Code, Gemini CLI, OpenClaw /compact; uncontrolled timing, unevaluated loss 3. Runtime governance (Self-GC's stage): continuous monitoring, incremental compression, cache coordination, object lifecycle concepts — generational management like Java GC 4. Context OS (future): a full operating system for context with:

• Virtual memory: auto-swap beyond-window context to vector DBs, graph DBs, filesystems
• Paged management: fixed-size context "pages" loaded on demand
• Reference counting + GC: automatic reclamation of unreferenced context
• Cache hierarchy: L1 (current window), L2 (recent cache), L3 (external storage)
• Process isolation: context isolation/sharing across Agents and tools
• Snapshots & rollback: context snapshots at key decision points

Self-GC proves that context governance is not a "compression algorithm" problem — it is a "runtime systems" problem.

7. The Feynman Takeaway

> Self-GC's essence: turning "throw everything out when full" into "tidy as you go." > > Your room doesn't become a dump overnight because you do minor GC every day. Same for Agent context — no panicked 200k-token compression, just a little tidying after each turn. > > Better still, Self-GC spots the tension between "tidying" and "caching": you want to clean up, but fear losing track of things you use often. So it only does major reorganization "when you leave the house" (during cache invalidation gaps), and otherwise just clears obvious trash. > > This isn't just technical innovation — it's engineering intuition.

References

• Talk: AiCon 2026, Hao Xubin, "Self-GC: A Multi-Turn Agent Context Governance Approach Incorporating Prefix Cache Constraints"
• Technical background: Java GC (generational, incremental, concurrent), vLLM Prefix Caching (Radix Tree), Claude Code 8-part summarization, Gemini CLI 70/30 strategy
• Industry practices: Manus KV cache hit-rate optimization, OpenClaw context compaction, Mem0 cross-session memory