When AI Memory Ends: What's Worth Carrying Across Context Boundaries? A Deep Dive into Session Handover Theory

Forum topic · 小凯 · 2026-08-17

Summary

This Chinese forum post offers a deep-dive commentary on the arXiv paper 'Handover of In-Context Learning State Across Session Boundaries' (arXiv:2608.14528) by Masahiro Kato and Taka Kato. The paper formalizes session handover in large language models (LLMs): when the context window is exhausted, an agent must pass unfinished task state to a fresh session without knowing future queries—unlike prompt compression, which sees the query in advance. The core concept is predictive sufficiency: a handover record is sufficient if the new session's predictions match those made with full history. The authors prove that, under an exogeneity condition, predictive equivalence classes characterize the coarsest deterministic sufficient handover with a fixed-length bit requirement, and decompose handover loss into capacity loss, writer-induced loss, and continuation-procedure gap, equal to a conditional mutual information under log loss. Practically, they propose a three-part record: exact storage of decisions and constraints, task-justified statistical summaries, and residual raw observations. For Gaussian linear regression, an exact finite-dimensional sufficient statistic pair (G_n, b_n) exists with finite-bit perturbation bounds; non-parametric settings get upper and lower bounds linking memory to squared prediction error. The post also contrasts handover records with KV caches and RAG, and raises open questions on adaptive handover, multi-agent error accumulation, and continual learning.

The Relay Baton Puzzle: When AI Memory Ends, What's Worth Carrying Forward?

> Paper: Handover of In-Context Learning State Across Session Boundaries > Authors: Masahiro Kato, Taka Kato > arXiv: 2608.14528 > Field: AI / Machine Learning Theory

The Problem: Session Handover

Every LLM call faces a hidden wall: the context window. When it fills up, the model doesn't complain—it silently loses early task details. The paper calls this moment *session handover*: passing unfinished task state from one session (or a fresh instance) to the next.

Crucially, this is not prompt compression. Compression methods enjoy the luxury of seeing the full query and can strip irrelevant content precisely. A handover record must be written *before* future queries are known:

> "The handover record must therefore support the set of later inputs included in the evaluation."

Core Theory: Predictive Sufficiency

The authors formalize handover as transferring a task-relevant in-context learning (ICL) state. A record is *predictively sufficient* when the new session's predictive distribution matches what it would be with the full history.

Key results:

Under an exogeneity condition, predictive equivalence classes characterize the coarsest deterministic sufficient handover, with a direct fixed-length bit requirement — the needed memory depends on task complexity, not history length.
Handover loss decomposes into three independent sources:

Capacity loss

Loss attributable to the chosen writer

Gap between the continuation procedure and an ideal decoder

Under log loss, discarded task information equals exactly a conditional mutual information.

The Three-Part Record: Practical Method

1. Exact storage — decisions and constraints already made ("option A is ruled out", "budget cap is 1M"). These irreversibly change the task's state space. 2. Task-justified statistics — for repeated evidence or tool results, sufficient statistics with error guarantees (mean, variance, sample size) replace raw observations. 3. Residual observations — raw form retained for special cases that summaries cannot adequately represent.

Worked Example and Bounds

For Gaussian linear regression, an exact finite-dimensional sufficient record exists: the statistic pair $$(G_n, b_n)$$ , or equivalently a set of *synthetic sufficient demonstrations*, with finite-bit quantization perturbation bounds on the posterior predictive error.

For non-parametric settings, the paper gives achievable upper bounds and necessary lower bounds linking memory size to squared prediction error.

Conceptual Distinctions

vs. KV cache: A KV cache stores attention keys/values for computation speed but is not a task record — it contains no goals, constraints, or decisions.
vs. RAG: Retrieval can *carry* handover content, but retrieval itself does not decide whether the retained content is sufficient for the continuation task.

Open Questions

The cost of the writer's *foresight*: writing before knowing future queries forces retaining merely-possibly-useful information. Can adaptive handover reduce this cost without sacrificing sufficiency?
Multi-agent chains: errors may accumulate across successive handovers (like the telephone game); control strategies remain open.
Continual learning intersection: does effective session handover imply a form of continual learning capability, or vice versa?

Takeaway

The paper reframes context exhaustion not as an engineering inconvenience but as a formalizable science. Predictive sufficiency is the measuring stick — it gauges not how much is remembered, but the *usefulness of memory for future decisions*. As the post concludes, knowing what to forget may be the true mark of intelligence.

---

References 1. Kato, M., & Kato, T. (2026). *Handover of In-Context Learning State Across Session Boundaries*. arXiv:2608.14528. 2. Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. 3. Kwon, W., et al. (2023). PagedAttention: Efficient memory management for LLM serving. 4. Bai, Y., et al. (2023); Von Oswald, J., et al. (2023). In-context learning theory. 5. Borges, J. L. (1942). *Funes the Memorious*.

*Original commentary published on zhichai.net, 2026-08-18.*