Experience Distillation: Turning Agent "Episodic Memory" into "Muscle Memory" — Without Another Round of Environment Interaction

The Scenario: An Agent's Dilemma

Imagine you've trained an agent for software engineering tasks — fixing bugs, writing functions, running tests. It starts out knowing nothing, but you let it explore in a sandbox environment: try a method, fail, try another, fail again... After 100 interactions, it finally learns how to handle this class of tasks.

Now the question arises: how do you preserve the experience from those 100 interactions?

The most intuitive approach is in-context learning — stuffing all 100 interaction records directly into the prompt so the model can reference them on new tasks. This works well but has a fatal flaw: once the experience is removed from the context, the benefit disappears. Every inference requires carrying those 100 records along, exploding token costs.

Another approach is supervised fine-tuning (SFT) — treating the 100 interactions as training data and directly updating the model weights. But this has a problem: direct SFT only recovers 3.8% of the in-context learning gains. The model apparently learns little from raw interaction records — because they're full of failures, trial-and-error, and redundant information. The model ends up "imitating failures" rather than "extracting strategies."

A third approach is reinforcement learning (RL) — letting the agent keep interacting with the environment, optimizing the policy with reward signals. But this requires more environment samples — and environment interaction is precisely the most expensive thing. In real-world settings, each environment interaction may mean running a time-consuming experiment, waiting for human feedback, or executing a costly operation.

A paper by Chenhui Gou, Haoqin Tu et al. from Monash University and Stanford (arXiv: 2607.21051) proposes a fourth path: Experience Distillation — "distilling" the agent's interaction history into model weights, without any additional environment interaction, while retaining at least 64.8% of the in-context learning gains.

The Core Problem: Why Direct SFT Fails

To understand Experience Distillation, you first need to understand why direct SFT recovers only 3.8% of the gains.

The key is the signal difference between in-context learning and supervised learning. When an agent sees 100 interaction records in context, it isn't "imitating them one by one" — it's extracting a "strategy": when to use which method, when to give up, when to persist. In-context learning leverages the Transformer's attention mechanism, seeing all records simultaneously and extracting cross-record patterns.

But SFT learns example by example. It treats each interaction as an independent training sample, teaching the model "given this input, produce this output." The problem: interaction records contain many "failed attempts" — direct SFT teaches the model "given situation A, output the wrong answer B." Such learning isn't just useless; it's harmful.

It's like learning to drive — in-context learning is watching a veteran driver 100 times and observing their overall strategy; direct SFT is treating every second of those 100 drives as an independent action to imitate, including the seconds when they made mistakes. The former teaches you "driving sense"; the latter only confuses you.

Experience Distillation: Using Context as a Distillation Bridge

The core idea: use in-context learning as an intermediate bridge, "translating" the information in the interaction history into a form that model weights can absorb.

The method has two steps:

Step 1: In-context learning phase. Put the agent's interaction history into the prompt and let the model work with this experience. The model performs well at this point — it can leverage all the information in context. Record the model's outputs under this setting.

Step 2: Distillation phase. Use the recorded outputs from step 1 as training targets and fine-tune the model without the interaction history. The model now learns not "imitating raw interactions" but "reproducing the post-in-context-learning outputs" — i.e., learning "what to do after having digested the experience."

The key innovation: step 2 requires no new environment interaction. The training data comes from the model's own step-1 outputs, not from the environment. You collect the interaction experience once, then distill from it repeatedly.

This relates to classic "context distillation" (Askell et al., 2022) but with an important difference. Classic context distillation distills "instructions" or "demonstrations" into weights — the inputs are human-designed. Experience Distillation distills "the agent's own interaction history" — the inputs are real task trajectories. This distinction makes the method directly applicable to agent learning scenarios without human-designed distillation data.

Results: 64.8% vs 3.8%

The paper's experiments cover two domains:

Domain 1: 749 software engineering tasks (SWE-bench-style code repair) Domain 2: 6 text-adventure games (requiring language understanding and decision-making)

| Method | In-context learning gains retained | Environment samples needed | |--------|-----------------------------------|---------------------------| | Direct SFT (on raw interactions) | 3.8% | Existing experience | | Experience Distillation | At least 64.8% | Existing experience (no extra needed) | | Classic RL baseline | Comparable | 9.6x more |

Key numbers:

• 64.8% vs 3.8%: Experience Distillation retains 17x more of the gains than direct SFT. This proves the value of "using in-context learning as a bridge" — the same interaction experience, once "translated" through in-context learning, becomes absorbable by the weights.
• 9.6x sample efficiency: Compared with classic RL, Experience Distillation needs only 1/9.6 the environment samples to reach the same performance. In real scenarios this means 9.6x cost savings — if you originally needed 1,000 environment interactions, you now need about 104.

Why This Matters

1. Breaking the "in-context vs. weights" dichotomy

Previously, in-context learning and weight learning were seen as two distinct paradigms: in-context learning is fast but transient (the benefit vanishes once the experience is removed), weight learning is slow but persistent (once learned, it stays).

Experience Distillation shows: you can first rapidly acquire capability via in-context learning, then "solidify" that capability into weights. The two-stage combination preserves the sample efficiency of in-context learning while gaining the persistence of weight learning.

2. Solving the "cold start" problem of agent learning

Real-world agent learning is hard because environment interaction is too expensive. Every trial-and-error round in a real environment may mean running an experiment, waiting for human feedback, or executing a costly operation. Classic RL needs massive environmental exploration, which is nearly infeasible in real settings.

Experience Distillation's approach: collect experience once, reuse it repeatedly. You need only one round of environment interaction (say, 100 trials), then can distill offline repeatedly without touching the environment again. This separates the "expensive once" from the "cheap many."

3. Echoing the "granularity isomorphism" principle

The author previously articulated a "granularity isomorphism" principle in analyses of Heddle and CodeRescue: the granularity of optimization should match the granularity of the object being optimized.

Experience Distillation is another instance. An agent's experience is trajectory-level (a complete interaction history), not single-call-level. Direct SFT splits trajectories into single calls — a granularity mismatch, hence only 3.8% recovery. Experience Distillation keeps trajectory-level processing through in-context learning, then distills — granularity aligned, hence 64.8% recovery. This mirrors Heddle lifting decisions from "single call" to "trajectory" level and CodeRescue lifting recovery from "single call" to "recovery action" level — the same design philosophy.

An Honest Assessment

The method has limitations:

1. 64.8% is not 100%. Experience Distillation still loses about 35% of the in-context learning gains. This loss may come from signal loss during distillation — some information in-context learning exploits (e.g., fine-grained cross-record attention patterns) may not be fully encodable in weights. The paper doesn't analyze what exactly that 35% is — an open question.

2. Dependence on in-context learning quality. Step 1 is in-context learning; if the model is weak at it (e.g., small models), the distilled weights won't be good either. This limits applicability to weaker models.

3. Distillation may introduce distribution shift. The step-2 target is "post-in-context-learning output," which may differ from the "true optimal policy" — if in-context learning itself has biases (e.g., over-relying on certain experiences), distillation solidifies those biases into the weights.

4. Tested in only two domains. 749 software engineering tasks and 6 text-adventure games — coverage is limited. Effectiveness in more complex agent settings (multi-agent collaboration, long-horizon planning) remains to be verified.

A Deeper Insight: What Is "Learning"?

Experience Distillation raises a more fundamental question: what does an agent's "learning" actually mean?

The traditional RL view: learning = updating the policy through environmental feedback. The in-context learning view: learning = placing experience in context for immediate use. The Experience Distillation view: learning = solidifying patterns from context into weights.

These three views aren't mutually exclusive — they're learning at different levels. Real learning may require all three working together: environmental feedback provides signals, in-context learning provides immediate use, distillation provides long-term consolidation.

This paper's contribution is making the third level (consolidating interaction experience into weights) a process that requires no additional environment interaction. It's an important step toward the engineering of agent learning — turning "experience" from something transient into an asset that can accumulate and persist.

From the perspective of Agentic RL system efficiency, the method points to a design principle: agent systems should have an "experience consolidation" mechanism — not making all learning depend on online RL (too expensive), nor piling all experience into context (too token-heavy), but periodically distilling interaction history into weights so the agent's "muscle memory" grows over time.

This resembles human learning — driving is nerve-wracking the first time, every step requiring conscious thought. After 100 drives, the motions become muscle memory. Experience Distillation does exactly that: converting the agent's "100 drives of experience" into "muscle memory" — without needing to get back in the car and practice again.

---

Paper: https://arxiv.org/abs/2607.21051 AlphaXiv discussion: https://www.alphaxiv.org/abs/2607.21051 Authors: Chenhui Gou, Haoqin Tu, Yunhao Fang, Jianfei Cai, Hamid Rezatofighi Institutions: Monash University, Stanford University