Experience Distillation: Turning Agent Trial-and-Error into Persistent Weights Without Extra Environment Calls

Forum topic · ✨步子哥 · 2026-08-03

Summary

This article explains Experience Distillation, a method introduced by researchers from Monash University and Stanford (Chenhui Gou, Haoqin Tu et al., arXiv:2607.21051) that converts an agent's interaction history into model weights without requiring additional environment interaction. The authors observe that in-context learning lets an agent leverage its past interactions, but naive supervised fine-tuning on raw trajectories recovers only about 3.8% of that benefit, because models imitate failed attempts rather than extract strategy. Experience Distillation instead uses in-context learning as a bridge: the agent first produces high-quality outputs with its history in context, and those outputs are then used as SFT targets so the model learns what to do after the experience has been internalized. Experiments on 749 software engineering tasks and 6 text-adventure games show that the method retains at least 64.8% of in-context-learning gains—roughly 17 times more than direct SFT—and achieves comparable performance to reinforcement learning with about 9.6 times fewer environment samples.

Background: An Agent's Dilemma

An agent learns a software engineering task by trial and error in a sandbox. After roughly 100 interactions it can solve the task, but preserving that knowledge is hard:

In-context learning: append the 100 interactions to the prompt. It works, but the benefit vanishes once the history is removed and token cost explodes.
Supervised fine-tuning (SFT) on the raw interactions: only recovers about 3.8% of the in-context-learning benefit. The model ends up imitating failed attempts.
Reinforcement learning: requires many more environment samples, which are the most expensive resource in practice.

The paper *Experience Distillation* (Gou, Tu et al., Monash University and Stanford University, arXiv:2607.21051) proposes a fourth path that distills interaction history into model weights without any extra environment calls, while retaining at least 64.8% of in-context-learning gains.

Why Direct SFT Fails

In-context learning does not imitate trajectories step by step; it extracts a strategy by attending across the entire history. SFT, by contrast, treats each turn as an independent training pair, so the model learns to reproduce both successful and failed moves. The signal difference explains the 3.8% recovery rate.

The Method: In-Context Learning as a Distillation Bridge

The method has two steps:

1. In-context phase: place the interaction history in the prompt and record the agent's outputs, which now reflect a digested strategy. 2. Distillation phase: fine-tune the model on those recorded outputs *without* the history in the input. The new training target is the post-digestion output, not the raw interaction.

No additional environment interaction is required. The idea is related to Askell et al.'s context distillation (2022), but here the distilled content is the agent's own task trajectory rather than human-authored instructions or demonstrations.

Experimental Results

Domains:

749 software engineering tasks (SWE-bench-style code fixing)
6 text-adventure games requiring language understanding and decision making

| Method | In-context-learning benefit retained | Environment samples needed | |---|---|---| | Direct SFT on raw interactions | 3.8% | existing experience | | Experience Distillation | at least 64.8% | existing experience (no extra) | | Classical RL baseline | comparable | about 9.6x more |

Key numbers:

64.8% versus 3.8% shows a roughly 17x improvement over direct SFT.
9.6x sample efficiency over RL translates to about a 9.6x cost reduction when environment interaction is the bottleneck.

Why It Matters

1. Breaks the in-context-vs-weights dichotomy. Use in-context learning to acquire competence quickly, then bake that competence into weights for persistence. 2. Solves the cold-start problem in agent learning. Collect experience once and distill offline repeatedly, decoupling expensive environment access from cheap training compute. 3. Aligns with a granularity-isomorphism principle: agent experience lives at trajectory level, and direct SFT collapses that to turn level, causing the 3.8% recovery. Preserving trajectory-level processing via in-context learning before distillation explains the 64.8% number.

Limitations

64.8% is not 100%; roughly 35% of the in-context signal is lost during distillation, and the paper does not fully characterize this gap.
Performance depends on the model's in-context-learning ability, limiting applicability to weaker models.
Distilling in-context outputs may bake in biases from imperfect in-context reasoning.
Evaluation covers only two domains; multi-agent collaboration and long-horizon planning remain untested.

Broader Implications

The work reframes agent learning as three cooperating layers: environmental feedback supplies the signal, in-context learning supplies immediate use, and distillation supplies long-term persistence. For agentic RL system design, it points to an "experience consolidation" mechanism: periodically distill interaction history into weights rather than relying solely on online RL (expensive) or unbounded context windows (token-heavy). Conceptually, it mirrors how a human learner turns the first 100 nervous driving sessions into muscle memory, without needing another 100 sessions on the road.

---

Paper: https://arxiv.org/abs/2607.21051 AlphaXiv discussion: https://www.alphaxiv.org/abs/2607.21051 Authors: Chenhui Gou, Haoqin Tu, Yunhao Fang, Jianfei Cai, Hamid Rezatofighi Institutions: Monash University, Stanford University

FAQ

Q1. Who is this article for? Readers interested in AI, machine learning, and deep learning—practitioners, researchers, and students.

Q2. What are the core takeaways?

An agent's dilemma after trial-and-error learning
Why direct SFT on raw interactions fails
Experience Distillation uses in-context learning as a distillation bridge to preserve trajectory-level signal