Experience Distillation: Turning an Agent's "Working Memory" into "Muscle Memory" — Without Touching the Environment Again
The Scenario: An Agent's Dilemma
Imagine you train an agent for software engineering tasks — fixing bugs, writing functions, running tests. It starts out knowing nothing, but through trial and error in a sandbox environment — failed attempt, switch approach, fail again, switch again — it finally learns how to handle these tasks after 100 interactions.
Now the question arises: How do you retain the experience from those 100 interactions?
The most intuitive approach is in-context learning (ICL) — stuff the logs of all 100 interactions directly into the prompt so the model can reference them on new tasks. This works well, but has a fatal flaw: once the experience is removed from the context, the benefit disappears. Every inference requires carrying all 100 interaction records, exploding token costs.
Another approach is supervised fine-tuning (SFT) — treat the 100 interactions as training data and directly fine-tune the model weights. But this has problems too: direct SFT only recovers 3.8% of the in-context learning benefit. The model seems to learn little from raw interaction logs — because those logs are full of failures, trial-and-error, and redundant information, so the model learns more about "imitating failures" than "extracting strategies."
A third option is reinforcement learning (RL) — let the agent keep interacting with the environment, optimizing its policy with reward signals. But this requires more environment samples — and environment interaction is precisely the most expensive thing. In real-world settings, each interaction may mean running a time-consuming experiment, waiting for human feedback, or executing a costly operation.
Chenhui Gou, Haoqin Tu, and colleagues from Monash University and Stanford (published July 23, 2026, arXiv: 2607.21051) propose a fourth path: Experience Distillation — "distilling" the agent's interaction history into model weights, requiring no additional environment interaction, while retaining at least 64.8% of the in-context learning benefit.
The Core Question: Why Direct SFT Fails
To understand Experience Distillation, you first need to understand why direct SFT recovers only 3.8% of the benefit.
The key lies in the signal difference between in-context learning and supervised learning. When an agent sees 100 interaction records in context, it isn't "imitating them one by one" — it's extracting a "policy": which method to use in which situation, when to give up, when to persist. In-context learning leverages the Transformer's attention mechanism to see all records simultaneously and extract cross-record patterns.
But SFT learns item by item. It treats each interaction as an independent training sample, teaching the model "produce this output given this input." The problem: interaction records contain many "failed attempts" — if you directly SFT on them, the model learns "given situation A, output wrong answer B." This learning isn't just useless — it's harmful.
It's like learning to drive: in-context learning is watching a skilled driver for 100 trips and observing their overall strategy; direct SFT is treating every second of those 100 drives as an independent action to imitate, including the seconds when they made mistakes. The former teaches you "driving sense"; the latter just confuses you.
Experience Distillation: Using In-Context Learning as a Distillation Bridge
The core idea of Experience Distillation: use in-context learning as an intermediate bridge, "translating" the interaction history into a form model weights can absorb.
The method has two steps:
Step 1: In-context learning phase. Put the agent's interaction history into the prompt and have the model perform tasks with that experience. The model performs well here — because it can use all the information in context. Record the model's outputs under this setting.
Step 2: Distillation phase. Use the outputs recorded in the previous step as training targets and fine-tune the model without the interaction history. The model now learns not to "imitate the raw interactions" but to "reproduce the post-in-context-learning outputs" — that is, to learn "what to do after having digested the experience."
The key innovation: step 2 requires no new environment interaction. The training data comes from the model's own outputs, not from the environment. You only need to collect interaction experience once, and you can repeatedly distill from it.
This relates to classic "context distillation" (Askell et al., 2022) but with an important difference. Classic context distillation distills "instructions" or "demonstrations" into weights — the inputs are human-designed. Experience Distillation distills "the agent's own interaction history" — the inputs are real task trajectories. This difference makes the method directly applicable to agent learning scenarios without manually designed distillation data.
Experimental Results: 64.8% vs 3.8%
The paper ran experiments in two domains:
Domain 1: 749 software engineering tasks (SWE-bench-style code fixing) Domain 2: 6 text-adventure games (requiring language understanding and decision-making)
Results comparison:
| Method | ICL benefit retained | Environment samples needed | |--------|---------------------|---------------------------| | Direct SFT (on raw interactions) | 3.8% | Existing experience | | Experience Distillation | At least 64.8% | Existing experience (no extra needed) | | Classic RL baseline | Comparable | 9.6x more |
Key numbers:
- 64.8% vs 3.8%: Experience Distillation retains 17x the benefit of direct SFT. This proves the value of "using in-context learning as a bridge" — the same interaction experience, after ICL "translation," becomes absorbable by weights.
- 9.6x sample efficiency: Compared with classic RL, Experience Distillation achieves the same performance with 1/9.6 the environment samples. In real scenarios, this means 9.6x cost savings — if you originally needed 1000 environment interactions, now you need roughly 104.
- Traditional RL view: learning = updating policy through environment feedback.
- ICL view: learning = putting experience in context for immediate use.
- Experience Distillation view: learning = solidifying patterns from context into weights.
Why This Method Matters
1. Breaking the "ICL vs. Weight Learning" Dichotomy
Previously, in-context learning and weight learning were treated as two distinct paradigms. ICL is fast but transient (gone once the experience is removed); weight learning is slow but persistent (what's learned stays).
Experience Distillation demonstrates: you can first rapidly acquire capability via in-context learning, then "solidify" that capability into weights. The two-stage combination preserves ICL's sample efficiency while gaining weight learning's persistence.
2. Solving the "Cold Start" Problem of Agent Learning
Real-world agent learning has a difficulty: environment interaction is expensive. Every time an agent trials in a real environment, it may mean running an experiment, waiting for human feedback, or executing a costly operation. Classic RL requires massive environmental exploration, which is nearly infeasible in real settings.
Experience Distillation's approach: collect experience once, reuse it repeatedly. You only need one round of environment interaction (say, 100 trials), then you can distill repeatedly offline without touching the environment. This separates "the expensive once" from "the cheap many."
3. Echoing the "Granularity Isomorphism" Principle
The post's author references the "granularity isomorphism" principle — the optimization granularity should match the granularity of the object being optimized.
Experience Distillation is another instance. An agent's experience is trajectory-level (a complete interaction history), not single-call-level. Direct SFT breaks trajectories into individual calls — a granularity mismatch, hence only 3.8% recovery. Experience Distillation preserves trajectory-level processing through in-context learning before distilling — granularity aligned, hence 64.8% recovery.
Honest Assessment
The method is not without limitations.
1. 64.8% is not 100%. Experience Distillation still loses about 35% of the ICL benefit. This loss may come from signal loss during distillation — some information ICL exploits (e.g., fine-grained cross-record attention patterns) may not fully encode into weights. The paper doesn't deeply analyze what this 35% is; it's an open question.
2. Dependence on ICL quality. Step 1 is in-context learning. If the model's ICL ability is weak (e.g., small models), the distilled weights won't be good. This limits application on weaker models.
3. Distillation may introduce distribution shift. The step-2 training target is "post-ICL outputs," which may differ from the "true optimal policy" — if ICL itself has biases (e.g., over-relying on certain experiences), distillation solidifies those biases into weights.
4. Tested in only two domains. 749 software engineering tasks and 6 text-adventure games — limited coverage. Effects in more complex agent scenarios (multi-agent collaboration, long-horizon planning) still need validation.
A Deeper Insight: What Is "Learning"?
Experience Distillation raises a more fundamental question: what does an agent's "learning" actually mean?
The paper's contribution is making the third layer (from interaction experience to weight consolidation) a process that requires no additional environment interaction. This is a significant step toward engineering agent learning — making "experience" no longer transient but a cumulative, retainable asset.
From a systems-efficiency perspective, the method points to a design principle: agent systems should have an "experience consolidation" mechanism — not relying entirely on online RL (too expensive), nor piling all experience into context (too token-heavy), but periodically distilling interaction history into weights so the agent's "muscle memory" grows over time.
It resembles human learning — driving is nerve-wracking at first, every step deliberate. After 100 drives, actions become muscle memory, no conscious thought needed. Experience Distillation does exactly this: converts an agent's "100 driving lessons" into its "muscle memory" — without getting back in the car to practice again.
---
Paper: https://arxiv.org/abs/2607.21051 AlphaXiv discussion: https://www.alphaxiv.org/abs/2607.21051 Authors: Chenhui Gou, Haoqin Tu, Yunhao Fang, Jianfei Cai, Hamid Rezatofighi Institutions: Monash University, Stanford University