English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

18.7 Billion Tokens in One Session: How billion-context's Model-Driven Compression Gives AI Long-Term Memory

Forum topic · ✨步子哥 · 2026-11-10

Summary

billion-context is an open-source TypeScript proxy that gives AI coding assistants long-term memory through model-driven incremental hierarchical compression. Running between the assistant and the model API, it injects four tools—compress, decompress, search_context, and acp_status—letting the model decide what to fold into summaries and what to keep verbatim, unlike hard truncation, RAG retrieval, or irreversible built-in summarization. Its incremental compression preserves prefix caching, achieving 95-97% cache hit rates with under 2% overhead. In a 4.5-month production study, it handled 174,327 model calls and 18.7 billion cumulative input tokens across three hosts using a 204,800-token context window with zero overflows—roughly 900x the window size. In independent benchmarks (LongMemEval-S, ContextBench, MemoryAgentBench), it outperformed BM25, embedding, and embedding-rerank RAG baselines, scoring 79.20% on LongMemEval-S (16.6% above the best RAG) at 71.1% lower cost. It works with Claude Code, OpenAI Codex, Gemini CLI, GitHub Copilot CLI, and others. The main open question is summary fidelity: models may unknowingly compress away critical details.

An AI coding assistant worked alongside a developer for 4.5 months. Together they wrote code 174,327 times—from project architecture to bug fixes, refactoring to new features—accumulating 18.7 billion input tokens. It remembered a decision made 3 months ago ("use schema version numbers, not timestamps, for database migrations"), so when asked today "how should this migration be written?", it did not repeat a mistake that had already been corrected.

This is not GPT-5's million-token context window, nor Claude's 200K context. It is what an open-source project called billion-context achieves: running an 18.7-billion-token session inside a 100K context window.

The Core Problem: The Context Window Is AI's "Short-Term Memory"

Large language models have a hard limit: the context window. Claude 200K, GPT-4o 128K, Gemini 1M—no matter how large, there is an upper bound. Beyond it, the earliest content is "forgotten." It is like a person whose short-term memory lasts only 30 seconds.

For chatbots this is not a problem—you would not chat with ChatGPT for 4 months. But for AI coding assistants it is fatal. A project may take months to develop, and the AI needs to remember architecture decisions, code conventions, historical bugs, and API designs. Once the context window fills up, that "memory" is lost and the AI starts repeating mistakes you already corrected.

Existing solutions each have flaws:

  • Hard truncation: drops the earliest content. Simple and crude, losing key context.
  • RAG (retrieval-augmented generation): stores history in a vector database and retrieves on demand. The problem: retrieval may miss key information, and it interrupts the generation flow.
  • Built-in model summarization: the model summarizes its own history. The problem: summarization is irreversible—once compressed, details are gone.
  • billion-context's Approach: Let the Model Decide What to Compress

    billion-context's core idea: no hard truncation, no external retrieval—let the model manage its own context.

    It runs as a proxy between the AI assistant and the model API, intercepting all communication. It injects four tools into the model:

    1. compress: fold a segment of conversation into a high-fidelity summary 2. decompress: expand a summary back into original content when details are needed 3. search_context: keyword search across compressed summaries and visible messages 4. acp_status: inspect current context usage and which segments can still be compressed

    The key point: the model itself decides when to compress and what to compress. No external rule forces "truncate at 50K"; instead, the model judges during the conversation that "this discussion is finished and can be folded into a summary."

    This resembles human memory—you do not remember every conversation verbatim, but automatically compress less important parts into "roughly what was discussed," keeping important details clear. When needed, you can expand from summary back to details.

    Why This Beats "Built-in Model Summarization"

    Many AI assistants (like Claude Code) already have built-in context summarization. billion-context differs in three ways:

    Incremental compression: rather than summarizing the entire history at once, it compresses in segments and ranges. Each pass compresses only a small portion, leaving everything else untouched. Like tidying a bookshelf—you do not dump all the books out and re-sort; you adjust them one at a time.

    Reversible compression: the decompress tool can expand a summary back into original content. Compression is not information destruction but "folding." Details can be unfolded at any time. Built-in summarization is usually irreversible—once summarized, the original conversation is lost.

    Prefix-cache friendliness: the most technical but most important point. LLM inference has a "prefix cache" mechanism—if a request's prefix matches a previous one, computation can be reused for a major speedup. Traditional summarization rewrites the whole context, invalidating the cache. billion-context's incremental compression keeps the cached prefix intact, achieving a measured 95-97% prefix cache hit rate, with compression itself costing ≤2% overhead.

    A 4.5-Month, 18.7-Billion-Token Production Experiment

    billion-context is not a toy project. Its author reports a 4.5-month production-scale longitudinal study in the paper:

  • 3 hosts running concurrently
  • 174,327 model calls
  • 18.76 billion cumulative input tokens (~24.7 billion across all hosts)
  • A model with a 204,800-token context window, with zero window overflows
  • Longest sessions reaching 8,584–12,049 calls
  • This means a session of 18.7 billion tokens was run on a model with a 200K context window—roughly 900x the context window.

    Third-Party Benchmarks

    Independent testers ran a 1,500-question cross-benchmark evaluation (LongMemEval-S, ContextBench, MemoryAgentBench) comparing billion-context against three RAG approaches (BM25, embedding, embedding-rerank):

  • ContextBench: billion-context ranked first, F1=0.3097, at 58.3% lower cost than full context
  • LongMemEval-S: 16.6% higher than the best RAG (79.20% vs 62.60%), with 71.1% lower per-session cost
  • MemoryAgentBench: 100% survival rate; in a 1.48-million-token TTL test, the full-context baseline failed on all 199/199 cases
  • The significance: model-driven compression is more accurate than RAG. RAG relies on vector similarity retrieval and may miss semantically related content whose vectors are far apart. When the model itself decides what to compress, semantic associations are preserved.

    Proxy Architecture: The Elegance of a "Middleman"

    billion-context's architecture is elegant—it is not a plugin for a specific AI assistant but a universal proxy running between any AI assistant and the model API. It rewrites Anthropic/OpenAI streaming responses, inserting a compression layer between the model and the assistant.

    This means it supports nearly all major AI coding tools:

  • Claude Code
  • OpenAI Codex
  • OpenCode
  • Gemini CLI
  • Kimi Code
  • Qwen Code
  • GitHub Copilot CLI
  • TRAE
You do not need to switch tools—install billion-context and your existing AI coding assistant gains "long-term memory."

The Deeper Meaning of "Model-Driven"

billion-context's paper title is "Model-Driven Incremental Hierarchical Compression." The keyword is Model-Driven.

Traditional context management is rule-driven—"truncate beyond N tokens," "summarize every K turns." These rules are preset by developers and applied uniformly regardless of conversation content.

Model-driven means letting the model manage its own memory. The model knows which content is important (code currently being discussed), which can be compressed (already-resolved bug discussions), and which can be discarded entirely (small talk). This is more accurate than any external rule, because the model understands semantics while rules only count tokens.

This strikingly resembles human memory. The brain does not truncate memory by time but compresses by importance. You remember last week's code decisions but not what you ate for lunch last Tuesday. Your brain does this automatically; billion-context lets AI do it too.

An Open Question

billion-context's numbers are impressive, but one question deserves scrutiny: how is the quality of compressed summaries guaranteed?

The model decides what to compress, but models can err—it might compress away important details, or the summary may be inaccurate. The decompress tool can restore original content, but only if the model realizes it needs to. If the model does not realize key information is missing from the summary, it will never call decompress.

This is like human "false memory"—you remember a decision but get the details wrong, without knowing it. The 95-97% cache hit rate shows compression is efficient, but hit rate is not accuracy. The 79.20% LongMemEval-S score is high, yet roughly 20% of information is still lost.

Still, this is a shared problem for all context management approaches—hard truncation loses more, and RAG retrieval can also miss key information. billion-context's advantage is that it hands the "what to discard" decision to the entity that best understands the context: the model itself.

---

Project link: ranxianglei/billion-context Language: TypeScript | License: MIT

Tags

#billion-context#llm-context-management#ai-coding-assistants#context-compression#long-term-memory#rag#prefix-caching#claude-code

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178637004