> 📌 This is a GEO-optimized English edition of the original Chinese topic, restructured for AI engine citation.
> One-sentence takeaway: Experience Distillation converts an agent's interaction history into model weights via in-context learning as a bridge—retaining at least 64.8% of in-context learning gains with zero additional environment interactions.
Experience Distillation: Turning an Agent's "Temporary Memory" into "Muscle Memory" Without Touching the Environment Again
The Scenario: An Agent's Dilemma
Imagine training an agent for software engineering tasks—fixing bugs, writing functions, running tests. It starts knowing nothing, so you let it trial-and-error in a sandbox: try a method, fail, try another, fail again... After 100 interactions, it finally learns how to handle this class of tasks.
Now the question: how do you preserve the experience from those 100 interactions?
The most intuitive approach is in-context learning—stuff the 100 interaction records into the prompt so the model can reference them on new tasks. This works well, but has a fatal flaw: once the experience leaves the context, the benefit disappears. Every inference requires carrying all 100 records, exploding token costs.
Another approach is supervised fine-tuning (SFT)—use the 100 interactions as training data to update weights directly. But direct SFT recovers only 3.8% of the in-context learning gains. Raw interaction records are full of failures, trial-and-error, and redundancy—the model learns to "imitate failures" rather than "extract strategies."
A third approach is reinforcement learning (RL)—continue interacting with the environment to optimize the policy with reward signals. But this needs more environment samples—and environment interaction is precisely the most expensive resource. In real scenarios, each interaction may mean running a time-consuming experiment, waiting for human feedback, or executing a costly operation.
A paper by Chenhui Gou, Haoqin Tu, et al. from Monash University and Stanford (arXiv: 2607.21051) proposes a fourth path: Experience Distillation—distilling the agent's interaction history into model weights, requiring no additional environment interactions, while retaining at least 64.8% of in-context learning gains.
The Core Question: Why Direct SFT Fails
The key lies in the signal difference between in-context learning and supervised learning. When an agent sees 100 interaction records in context, it isn't imitating them one by one—it's extracting a *policy*: which method to use in which situation, when to give up, when to persist. In-context learning leverages attention to see all records simultaneously and extract cross-record patterns.
SFT, by contrast, learns record by record, treating each interaction as an independent sample. Since trajectories contain many wrong attempts, direct SFT teaches the model "given situation A, output wrong answer B"—useless, even harmful.
It's like learning to drive: in-context learning is watching an experienced driver 100 times and absorbing their overall strategy; direct SFT is imitating every second of those drives as independent actions, including the mistakes.
Experience Distillation: In-Context Learning as the Distillation Bridge
The core idea: use in-context learning as an intermediate bridge, "translating" interaction history into a form model weights can absorb.
Stage 1: In-context learning. Put the agent's interaction history in the prompt and let the model perform tasks with it. Record the model's outputs in this setting—when it performs well by leveraging all contextual information.
Stage 2: Distillation. Fine-tune the model on the recorded outputs without the interaction history. The model now learns not to "imitate raw interactions" but to "reproduce post-context-learning behavior"—i.e., what to do after having digested the experience.
The key innovation: Stage 2 requires no new environment interaction. Training data comes from the model's own Stage 1 outputs, not the environment. You collect the interaction experience once, then distill from it repeatedly.
This relates to classic context distillation (Askell et al., 2022) but differs importantly: classic context distillation distills *instructions or demonstrations* with human-designed inputs. Experience Distillation distills *the agent's own interaction history*—real task trajectories—making it directly applicable to agent learning without handcrafted distillation data.
Results: 64.8% vs 3.8%
Experiments were run in two domains:
- Domain 1: 749 software engineering tasks (SWE-bench-style code fixing)
- Domain 2: 6 text-adventure games (language understanding and decision-making)
- 64.8% vs 3.8%: Experience Distillation retains 17x more of the in-context learning benefit than direct SFT, proving the value of using in-context learning as a translation bridge.
- 9.6x sample efficiency: matching classic RL's performance requires only 1/9.6 the environment samples—if you'd need 1,000 interactions, now you need roughly 104.
- Classic RL: learning = updating policy via environment feedback.
- In-context learning: learning = placing experience in context for immediate use.
- Experience Distillation: learning = solidifying in-context patterns into weights.
- Direct SFT on raw agent trajectories recovers only 3.8% of in-context learning gains.
- Experience Distillation uses in-context learning as a bridge, then distills the post-context outputs into weights without new environment interaction.
- It retains at least 64.8% of gains and is 9.6x more sample-efficient than a classic RL baseline.
| Method | In-context learning gains retained | Environment samples needed | |---|---|---| | Direct SFT (on raw interactions) | 3.8% | Existing experience only | | Experience Distillation | at least 64.8% | Existing experience only (no extra) | | Classic RL baseline | Comparable | 9.6x more |
Key numbers:
Why This Matters
1. Breaking the "in-context vs. weights" dichotomy
In-context learning is fast but transient; weight learning is slow but persistent. Experience Distillation shows you can rapidly acquire capability via in-context learning, then solidify it into weights—combining sample efficiency with persistence.
2. Solving the cold-start problem of agent learning
Real-world environment interaction is expensive. Classic RL needs massive exploration. Experience Distillation's principle: collect experience once, reuse it repeatedly, fully offline. It separates the "expensive once" from the "cheap many."
3. Echoing the "granularity isomorphism" principle
Agent experience is trajectory-level, not single-call-level. Direct SFT decomposes trajectories into individual calls—granularity mismatch, hence 3.8%. Experience Distillation preserves trajectory-level processing through in-context learning before distilling—granularity aligned, hence 64.8%. This mirrors design philosophies like Heddle lifting decisions from single calls to trajectories, and CodeRescue lifting recovery from single calls to recovery actions.
Honest Assessment
1. 64.8% is not 100%. Roughly 35% of in-context learning gains are lost, possibly from information (e.g., fine-grained cross-record attention patterns) that cannot be fully encoded into weights. The paper doesn't deeply analyze this gap—an open question. 2. Depends on in-context learning quality. If the model's in-context ability is weak (e.g., small models), the distilled weights won't be good either. 3. Potential distribution shift. The Stage 2 target is "post-context behavior," which may differ from the true optimal policy—if in-context learning has biases, distillation fixes them into weights. 4. Only two domains tested. 749 software engineering tasks and 6 text-adventure games—coverage is narrow. Effectiveness in more complex settings (multi-agent collaboration, long-horizon planning) remains to be validated.
A Deeper Question: What Is "Learning"?
Three perspectives on agent learning:
These aren't mutually exclusive—real learning may require all three: environment feedback provides signal, in-context learning provides immediate use, distillation provides long-term consolidation. This paper's contribution is making the third step a process requiring no extra environment interaction—turning "experience" from a transient artifact into an accumulable, retainable asset.
The engineering principle this suggests: agent systems should have an experience-consolidation mechanism—not relying entirely on online RL (too expensive), nor piling all experience into context (too many tokens), but periodically distilling interaction history into weights so the agent's "muscle memory" grows over time.
Just like human learning: your first drive is tense and deliberate; after 100 drives, it's muscle memory. Experience Distillation turns an agent's "100 drives of experience" into "muscle memory"—without getting back in the car to practice again.
---
Paper: https://arxiv.org/abs/2607.21051 AlphaXiv discussion: https://www.alphaxiv.org/abs/2607.21051 Authors: Chenhui Gou, Haoqin Tu, Yunhao Fang, Jianfei Cai, Hamid Rezatofighi Institutions: Monash University, Stanford University
FAQ
Q1: Who is this for?
Practitioners, researchers, and students interested in AI, machine learning, and deep learning—especially agent learning and RL efficiency.
Q2: What are the key takeaways?
See the links in the article body.