Experience Distillation: Turning an Agent's "Transient Memory" into "Muscle Memory" — Without Interacting with the Environment Again

The Scenario: An Agent's Dilemma

Imagine you train an agent for software engineering tasks — fixing bugs, writing functions, running tests. It starts out knowing nothing, so you let it explore a sandbox environment through trial and error: try a method, fail, switch, fail again, switch again... After 100 interactions, it finally learns how to handle this type of task.

Now the question arises: how do you preserve the experience from those 100 interactions?

The most intuitive approach is in-context learning — stuffing the records of all 100 interactions directly into the prompt so the model can reference them on new tasks. This works well but has a fatal flaw: once the experience is removed from the context, the benefit disappears. Every inference must carry those 100 interaction records, exploding token costs.

Another approach is supervised fine-tuning (SFT) — treating the 100 interactions as training data and directly fine-tuning model weights. But this has problems too: direct SFT recovers only 3.8% of the in-context learning benefit. The model seems unable to learn much from raw interaction logs — because the records are full of failures, trial-and-error, and redundant information. The model learns more about "imitating failure" than "extracting strategy."

A third approach is reinforcement learning (RL) — letting the agent keep interacting with the environment and optimizing the policy with reward signals. But this requires more environment samples — and environment interaction is exactly the most expensive thing. In real-world settings, each interaction may mean running a time-consuming experiment, waiting for human feedback, or executing a costly operation.

A paper by Chenhui Gou, Haoqin Tu et al. from Monash University and Stanford (arXiv: 2607.21051) proposes a fourth path: Experience Distillation — "distilling" the agent's interaction history into model weights, requiring no additional environment interaction, while retaining at least 64.8% of the in-context learning benefit.

The Core Question: Why Direct SFT Fails

To understand Experience Distillation, you first need to understand why direct SFT recovers only 3.8% of the benefit.

The key lies in the signal difference between in-context learning and supervised learning. When an agent sees 100 interaction records in context, it isn't "imitating item by item" — it's extracting a "policy": which method to use in which situation, when to give up, when to persist. In-context learning leverages the Transformer's attention mechanism, seeing all records simultaneously and extracting cross-record patterns.

But SFT learns item by item. It treats each interaction as an independent training sample, teaching the model "given this input, produce this output." The problem: interaction records contain many "failed attempts" — if you apply SFT directly, the model learns "when seeing situation A, output wrong answer B." Such learning is not just useless, it's harmful.

It's like learning to drive — in-context learning is watching a veteran driver for 100 trips and observing their overall strategy; direct SFT is treating every second of those 100 drives as an independent action to imitate, including the seconds when they made mistakes. The former teaches you a "feel for driving"; the latter only confuses you.

Experience Distillation: Using Context as a Distillation Bridge

The core idea: use in-context learning as an intermediate bridge, "translating" the information in interaction history into a form that model weights can absorb.

The method has two steps:

Step 1: In-context learning phase. Put the agent's interaction history into the prompt and let the model perform tasks with that experience. The model performs well here — it can use all the information in context. Record the model's outputs in this setting.

Step 2: Distillation phase. Use the outputs from Step 1 as training targets and fine-tune the model without the interaction history. The model now learns not to "imitate raw interactions" but to "reproduce the post-in-context-learning output" — i.e., to learn "what to do after having digested the interaction experience."

The key innovation: Step 2 requires no new environment interaction. The training data comes from the model's outputs in Step 1, not from the environment. You only need to collect interaction experience once, then reuse it for distillation repeatedly.

This relates to classic "context distillation" (Askell et al., 2022) but with an important difference. Classic context distillation distills "instructions" or "demonstrations" into weights — the input is human-designed. Experience Distillation distills "the agent's own interaction history" — the input is real task trajectories. This distinction makes the method directly applicable to agent learning scenarios without manually designed distillation data.

Experimental Results: 64.8% vs 3.8%

The paper ran experiments in two domains:

Domain 1: 749 software engineering tasks (SWE-bench-style code fixing tasks) Domain 2: 6 text-adventure games (requiring language understanding and decision-making)

Results comparison:

| Method | In-context learning benefit retained | Environment samples needed | |------|-------------------|-------------| | Direct SFT (on raw interactions) | 3.8% | Existing experience | | Experience Distillation | at least 64.8% | Existing experience (no extra needed) | | Classic RL baseline | Comparable | 9.6x more |

Key numbers:

• 64.8% vs 3.8%: Experience Distillation retains 17x more benefit than direct SFT. This proves the value of "using in-context learning as a bridge" — the same interaction experience, after being "translated" through in-context learning, becomes absorbable by weights.
• 9.6x sample efficiency: Compared to classic RL, Experience Distillation needs only 1/9.6 of the environment samples to reach the same performance. In real scenarios, this means 9.6x cost savings — if you originally needed 1000 environment interactions, you now need only about 104.

Why This Method Matters

1. Breaking the "In-Context Learning vs Weight Learning" Dichotomy

Previously, in-context learning and weight learning were viewed as two different learning paradigms. In-context learning is fast but transient (the benefit vanishes once the experience is removed), while weight learning is slow but persistent (once learned, it stays).

Experience Distillation demonstrates: you can first rapidly acquire capability through in-context learning, then "solidify" that capability into weights. The two-stage combination preserves the sample efficiency of in-context learning while gaining the persistence of weight learning.

2. Solving the "Cold Start" Problem of Agent Learning

Real-world agent learning faces a difficulty: environment interaction is too expensive. Every time an agent explores a real environment, it may mean running an experiment, waiting for human feedback, or executing a costly operation. Classic RL needs extensive environment interaction for exploration, which is nearly infeasible in real settings.

Experience Distillation's approach: collect experience once, reuse it repeatedly. You only need one round of environment interaction (say, 100 trials), then you can distill repeatedly offline without touching the environment again. This separates "the expensive once" from "the cheap many."

3. Echoing the "Granularity Isomorphism" Principle

The author previously derived a "granularity isomorphism" principle from analyses of Heddle and CodeRescue — the granularity of optimization should match the granularity of the thing being optimized.

Experience Distillation is another instance of this principle. An agent's experience is trajectory-level (a complete interaction history), not single-call-level. Direct SFT breaks trajectories into individual calls — a granularity mismatch, hence only 3.8% recovery. Experience Distillation keeps trajectory-level processing through in-context learning before distilling — granularity aligned, hence 64.8% recovery.

This is isomorphic to Heddle lifting decisions from "single-call" to "trajectory" level and CodeRescue lifting recovery from "single-call" to "recovery-action" level — the same design philosophy.

An Honest Assessment

The method is not without limitations.

1. 64.8% is not 100%. Experience Distillation still loses about 35% of the in-context learning benefit. This loss may come from signal loss during distillation — some information used by in-context learning (like fine-grained cross-record attention patterns) may not be fully encodable into weights. The paper doesn't deeply analyze what that 35% is — an open question.

2. Dependence on in-context learning quality. Step 1 is in-context learning; if the model's in-context learning ability is weak (e.g., small models), the distilled weights won't be good either. This limits application to weaker models.

3. Distillation may introduce distribution shift. The training target in Step 2 is "post-in-context-learning output," which may differ from "the true optimal policy" — if in-context learning itself has biases (e.g., over-relying on certain experiences), distillation will solidify those biases into weights.

4. Tested in only two domains. 749 software engineering tasks and 6 text-adventure games — insufficient coverage. How it performs in more complex agent scenarios (multi-agent collaboration, long-horizon planning) needs further validation.

A Deeper Insight: What Is "Learning"

Experience Distillation raises a more fundamental question: what does an agent's "learning" actually mean?

Traditional RL perspective: learning = updating the policy through environment feedback. In-context learning perspective: learning = putting experience into context for immediate use. Experience Distillation perspective: learning = solidifying patterns from context into weights.

These three perspectives are not mutually exclusive — they are learning at different levels. Real learning may require the coordination of all three: environment feedback provides signal, in-context learning provides immediate use, and distillation provides long-term solidification.

The paper's contribution is making the third level (solidifying interaction experience into weights) a process that requires no additional environment interaction. This is an important step in the engineering of agent learning — making "experience" no longer transient but an asset that can accumulate and be retained.

From the perspective of Agentic RL system efficiency, this method points to a design principle: agent systems should have an "experience solidification" mechanism — not relying entirely on online RL for learning (too expensive), nor piling all experience into context (too token-heavy), but periodically distilling interaction history into weights so the agent's "muscle memory" grows over time.

This resembles human learning — the first time you drive, you're nervous and think through every step. After 100 drives, the actions become muscle memory, no deliberate thinking needed. What Experience Distillation does is turn the agent's "100 driving experiences" into its "muscle memory" — without needing to get back in the car to practice again.

---

Paper link: https://arxiv.org/abs/2607.21051 AlphaXiv discussion: https://www.alphaxiv.org/abs/2607.21051 Authors: Chenhui Gou, Haoqin Tu, Yunhao Fang, Jianfei Cai, Hamid Rezatofighi Institutions: Monash University, Stanford University