REMEMORY: Learning a Residual to Recover What /compact Throws Away
Anyone working with long-horizon agents knows the picture: the context window keeps filling up, then you hit /compact — available in both Claude Code and Codex CLI — which compresses earlier interactions into a text summary, freeing the window so work can continue. Input length follows a sawtooth: grow, cut, grow again.
The cost of the sawtooth is hidden inside. A summary is a text checkpoint the agent resumes from, but a summary preserves the gist, and the gist is not everything: as the task unfolds, the relevance of earlier conversation segments changes. You don't know at compression time what you'll need later, so you can't preserve everything important in advance. Whether a plain-text summary can support every subsequent decision is a gamble.
A paper posted to arXiv on October 8 (REMEMORY, arXiv 2610.11287) offers a clean new solution: if the summary loses information, learn the lost part.
A Residual Connection Along the Sequence Dimension
REMEMORY is a separately trained memory network. Given the raw history and its corresponding summary, it outputs a bounded set of soft memory tokens that are appended after the summary. The training objective is direct: make the frozen LLM, conditioned on summary-plus-soft-tokens, produce output as close as possible to what it would produce given the full history.
In other words, the soft tokens learn not the history itself, but the difference between the summary and the history. The authors' name for it is therefore precise — residual memory: the summary flows along the sequence, and the soft tokens supply the correction that would otherwise be lost. Residual connections in deep learning let networks learn increments; this is the same idea, applied along the context dimension.
The LLM stays frozen throughout. Only the peripheral memory network changes — which is also why this kind of approach can follow any new model.
5.2% of the Positions, Near Full-Context Scores
One control in the evaluation deserves attention. Besides full context, summary-only, and summary-plus-soft-tokens, they added a group of "summary plus plain text of the same length" — as many ordinary text positions after the summary as the soft tokens occupy. This control answers one question: is the gain learned, or just a result of more positions?
The answer is: learned. On SummHay, a summarization attribution benchmark, the raw context averages 96,872 positions, the summary 1,393, and summary-plus-soft-tokens 5,024 — only 5.2% of the original. That 5.2% pulls the joint score back near full-context levels: citation F1 rises by 4.04 points, joint score by 3.55, insight coverage barely moves (0.40), and the paired bootstrap 95% interval is [0.74, 6.21]. The plain-text group with the same position budget performs almost identically to the summary alone. Positions don't solve the problem; the learned delta does.
The Agent Experience: No More Repeating Yourself
On end-to-end long-horizon agent tasks, two models run with residual memory: Qwen3.8-27B gains 9.8 points on AutomationBench and 7.6 on JobBench; Terminal-Bench gains 4.5 and BrowseComp 3.0. GLM-5.3-Flash (320B parameters) gains 3.3 and 4.1 respectively, pushed close to frontier-level performance. Both sets are averages over five runs.
More tangible than the scores is the behavioral change: agents with residual memory significantly reduce repeated outputs of the same tool results and repeated tool errors on BrowseComp and Terminal-Bench. This is the classic symptom of context loss — if the summary doesn't record "already checked this, the result was this," the agent checks again and errs again. Restoring the residual reduces these ineffective loops, and average API cost falls accordingly (the paper prices this at uncached September 2026 rates, counting soft-token positions as input; the memory network's own runtime overhead is not included — read the accounting that way).
The paper also includes a fine-grained analysis: from 100 real BrowseComp trajectories, they take 488 pairs of "next actions" after compression and score them pairwise with a judge model. The quality of the first action after compression is precisely the most fragile moment at the sawtooth's trough — and exactly where residual memory helps.
Attribution and Lineage
The author list is clear: the paper is led by the Shanghai Academy of AI for Science (SAIS) and Fudan University (both co-first authors and corresponding authors have SAIS emails), with participation from Shanghai Jiao Tong University, Michigan, CUHK, and Nanjing University; Alibaba is the sixth author's affiliation — video titles saying "proposed by Alibaba" should be read as: one of the collaborating institutions.
In the bigger picture: context compression is itself a lossy interface — the raw trajectory is replaced by a summary, and structure is lost at the compression step. Prior solutions in this family mostly aimed to "lose less" (better summaries, faithful storage). REMEMORY takes a different route: accept the loss, then treat the lost delta as a learnable target. It is complementary to approaches that store historical KV caches in a memory pool — one preserves faithfully and stays static, the other compresses hard and then patches back. And its best metaphor is the word itself: residual. Residual connections in networks let deep layers learn increments; residual memory lets an agent learn "what I forgot."