Experience Distillation: Turning "Temporary Memory" into "Muscle Memory" — Without Another Round of Environment Interaction
The Scenario: An Agent's Dilemma
Imagine you've trained an agent to do software engineering tasks — fixing bugs, writing functions, running tests. It starts out knowing nothing, but you let it explore a sandbox environment through trial and error: it tries one approach, fails, tries another, fails again... After 100 interactions, it finally learns how to handle this class of tasks.
Now the question: How do you preserve the experience from those 100 interactions?
The most intuitive approach is in-context learning — stuffing all 100 interaction records into the prompt so the model can reference them on new tasks. This works well, but has a fatal flaw: once the experience leaves the context, the benefit disappears. Every inference requires carrying all 100 interaction records, and token costs explode.
Another approach is supervised fine-tuning (SFT) — treating the 100 interactions as training data and directly fine-tuning the model weights. But this has problems too: direct SFT only recovers 3.8% of the in-context learning benefit. The model seems to learn little from raw interaction records — because those records are full of failures, trial-and-error, and redundant information. Instead of extracting strategy, the model mostly learns to "imitate failure."
A third approach is reinforcement learning (RL) — letting the agent keep interacting with the environment and optimizing with reward signals. But this requires many more environment samples — and environment interaction is precisely the most expensive thing. In real-world settings, each interaction might mean running a time-consuming experiment, waiting for human feedback, or executing a costly operation.
A paper from Monash University and Stanford (Chenhui Gou, Haoqin Tu et al., arXiv: 2607.21051) proposes a fourth path: Experience Distillation — distilling the agent's interaction history into model weights, without any additional environment interaction, while retaining at least 64.8% of the in-context learning benefit.
The Core Question: Why Direct SFT Fails
To understand Experience Distillation, you first need to understand why direct SFT only recovers 3.8% of the benefit.
The key is the signal difference between in-context learning and supervised learning. When an agent sees 100 interaction records in its context, it isn't imitating them one by one — it's extracting a *policy*: which method to use in which situation, when to give up, when to persist. In-context learning leverages the Transformer's attention mechanism to see all records simultaneously and extract cross-record patterns.
But SFT is example-by-example learning. It treats each interaction as an independent training sample, teaching the model "given this input, produce this output." The problem: interaction records contain many failed attempts — if you SFT on them directly, the model learns "when seeing situation A, output the wrong answer B." This learning isn't just useless; it's harmful.
It's like learning to drive: in-context learning is watching a skilled driver 100 times and absorbing their overall strategy; direct SFT is treating every second of those 100 drives as an independent action to imitate — including the seconds where the driver made mistakes. The former teaches you "driving sense"; the latter just confuses you.
Experience Distillation: In-Context Learning as the Distillation Bridge
The core idea: use in-context learning as an intermediate bridge, translating the interaction history into a form model weights can absorb.
The method has two steps:
Step 1: In-context learning phase. Put the agent's interaction history into the prompt and let the model perform tasks with that experience. The model performs well here because it can leverage all the contextual information. Record the model's outputs under this setting.
Step 2: Distillation phase. Use the outputs recorded in step 1 as training targets, and fine-tune the model without the interaction history. The model now learns not to "imitate raw interactions" but to "reproduce the post-in-context-learning behavior" — i.e., to internalize what to do *after* having digested the experience.
The key innovation: step 2 requires no new environment interaction. The training data comes from the model's own outputs, not from the environment. You collect interaction experience once, then reuse it for distillation repeatedly.
This relates to classic context distillation (Askell et al., 2022) but with an important difference. Classic context distillation distills "instructions" or "demonstrations" into weights — the inputs are human-designed. Experience Distillation distills the agent's own interaction history — the inputs are real task trajectories. This difference makes the method directly applicable to agent learning scenarios without requiring hand-crafted distillation data.
Experimental Results: 64.8% vs 3.8%
The paper evaluated the method in two domains:
- Domain 1: 749 software engineering tasks (SWE-bench-style code repair)
- Domain 2: 6 text-adventure games (requiring language understanding and decision-making)
- 64.8% vs 3.8%: Experience Distillation retains 17x more benefit than direct SFT. This proves the value of "using in-context learning as a bridge" — the same interaction experience, once "translated" through in-context learning, becomes absorbable into weights.
- 9.6x sample efficiency: Compared with classic RL, Experience Distillation achieves the same performance with 1/9.6 the environment samples. In real terms, that's a 9.6x cost saving — if you'd originally need 1,000 environment interactions, now you need about 104.
- Traditional RL view: learning = updating policy through environmental feedback.
- In-context learning view: learning = placing experience in context for immediate use.
- Experience Distillation view: learning = consolidating in-context patterns into weights.
Results comparison:
| Method | ICL Benefit Retained | Environment Samples Needed | |--------|---------------------|---------------------------| | Direct SFT (on raw interactions) | 3.8% | Existing experience only | | Experience Distillation | ≥ 64.8% | Existing experience only (no extra) | | Classic RL baseline | Comparable | 9.6x more |
Key numbers:
Why This Matters
1. Breaking the "in-context learning vs weight learning" dichotomy
Previous research treated in-context learning and weight learning as separate paradigms: in-context learning is fast but transient (gone when the experience leaves the context); weight learning is slow but persistent.
Experience Distillation shows you can first rapidly acquire capability through in-context learning, then consolidate it into weights. The two-stage combination keeps the sample efficiency of context learning while gaining the persistence of weight learning.
2. Solving the "cold start" problem in agent learning
A real-world difficulty in agent learning: environment interaction is too expensive. Letting an agent explore a real environment might mean running experiments, waiting for human feedback, or executing costly operations. Classic RL requires extensive environmental exploration, which is nearly infeasible in real settings.
Experience Distillation's approach: collect experience once, reuse it repeatedly. One round of environmental interaction (say, 100 trials), then offline distillation indefinitely — no more touching the environment. It separates the "expensive once" from the "cheap many."
3. Echoing the "granularity isomorphism" principle
The author notes that Experience Distillation instantiates a principle they call "granularity isomorphism": the optimization granularity should match the granularity of the object being optimized.
Agent experience is trajectory-level (a complete interaction history), not single-call-level. Direct SFT breaks trajectories into individual calls — a granularity mismatch, hence only 3.8% recovery. Experience Distillation maintains trajectory-level processing via in-context learning before distilling — granularity aligned, hence 64.8% recovery.
Honest Assessment
The method has real limitations:
1. 64.8% is not 100%. About 35% of the in-context learning benefit is still lost — possibly fine-grained cross-record attention patterns that can't be fully encoded into weights. The paper doesn't analyze what exactly is lost; this remains an open question.
2. Dependence on in-context learning quality. If the model's own in-context learning ability is weak (e.g., small models), the distilled weights won't be good either. This limits application on weaker models.
3. Distillation may introduce distribution shift. The step-2 target is "post-in-context-learning output," which may differ from the true optimal policy. If in-context learning itself has biases (e.g., over-relying on certain experiences), distillation bakes those biases into the weights.
4. Only two domains tested. 749 software engineering tasks and 6 text-adventure games — coverage is narrow. Performance in more complex agent scenarios (multi-agent collaboration, long-horizon planning) remains to be validated.
A Deeper Implication: What Is "Learning"?
Experience Distillation raises a more fundamental question: what does "learning" actually mean for an agent?
The paper's contribution is making the third layer (consolidating interaction experience into weights) a process that requires no extra environment interaction. That's a significant step toward engineering agent learning — turning "experience" from something transient into an asset that can accumulate and be retained.
From a system efficiency perspective, this points to a design principle: agent systems should have an "experience consolidation" mechanism — not relying entirely on online RL (too expensive), not piling all experience into context (too token-hungry), but periodically distilling interaction history into weights so the agent's "muscle memory" grows over time.
This resembles human learning: the first time you drive, every move takes conscious thought. After 100 drives, it becomes muscle memory. Experience Distillation does exactly this — converting an agent's "100 driving experiences" into "muscle memory" without ever getting back in the car.
---
Paper: https://arxiv.org/abs/2607.21051 AlphaXiv discussion: https://www.alphaxiv.org/abs/2607.21051 Authors: Chenhui Gou, Haoqin Tu, Yunhao Fang, Jianfei Cai, Hamid Rezatofighi Institutions: Monash University, Stanford University