SDAR: Self-Distilled Agentic Reinforcement Learning Tames Unstable Agent Training with Token-Level Trust Gating

Forum topic · 小凯 · 2026-05-17

Summary

SDAR (Self-Distilled Agentic Reinforcement Learning), proposed by researchers from Zhejiang University, Meituan, and Tsinghua University (arXiv:2605.15155), addresses a core problem in training LLM agents: trajectory-level reinforcement learning (GRPO) offers only coarse, end-of-episode rewards, while on-policy self-distillation (OPSD) from a privileged teacher branch becomes unreliable in multi-turn settings once the student diverges, causing KL divergence blow-ups and performance collapse. SDAR keeps GRPO as the primary optimization signal and demotes OPSD to an auxiliary role, gated per token. Three gating strategies are explored—entropy gating, teacher-student gap gating, and a soft-OR combination—with gap gating performing best, realizing asymmetric trust: strong supervision when the teacher endorses a token, softened supervision when it opposes. On ALFWorld, Search-QA, and WebShop with Qwen2.5/Qwen3 models (1.7B–7B), SDAR improves over GRPO by roughly 7–10 points (e.g., 84.4 vs. 75.0 on ALFWorld-3B; 53.9 vs. 46.1 on ALFWorld-1.7B), avoids the catastrophic instability of naive GRPO+OPSD, internalizes skills rather than depending on them at test time, and remains robust even with random skill retrieval.

SDAR: How a "Window of Trust" Tames the Runaway Spiral in Agent Training

> Self-Distilled Agentic Reinforcement Learning > arXiv: 2605.15155 | Zhengxi Lu et al., Zhejiang University & Meituan & Tsinghua > TL;DR: Turn the privileged teacher's chatter into a window that opens and closes automatically—amplifying guidance when it's trustworthy, muting it when it's not. That window is SDAR, boosting Qwen's performance on multi-turn agent tasks by up to 10%.

Introduction: A Parable of the Teacher in the Next Room

Imagine an escape-room game. You have a walkie-talkie to a "teacher" in the next room who sees everything you see—plus clues you don't have (e.g., "the third door is safe"). How do you use this teacher?

Option A: Ignore them entirely and grope in the dark. This is pure RL (GRPO): you only learn whether you ultimately escaped, never whether step 3 was correct.
Option B: Obey every instruction. But if the teacher's clues are sometimes wrong, stale, or based on a misread of your situation, blind obedience leads you astray. This is what happens when On-Policy Self-Distillation (OPSD) is naively stuffed into multi-turn agent training.

The team from Zhejiang University, Meituan, and Tsinghua proposed a third way: SDAR (Self-Distilled Agentic Reinforcement Learning). Instead of fully trusting or ignoring the teacher, SDAR installs a window with adjustable transparency between student and teacher—fully transparent when the teacher is right, fogged when the teacher may be wrong. Crucially, each token decides for itself how open the window should be.

Gate One: RL's Curse—Hindsight-Only Feedback

Modern LLM agents must complete complex tasks through multi-turn interaction: operating terminals, browsing the web, finding objects. The fatal problem with RL here is that rewards are trajectory-level—you learn only whether the journey succeeded, not which individual step was wrong. Like a coach holding a sign at the marathon finish line reading "4 hours" or "DNF," with no word about where you went off course at kilometer 15.

For long-horizon multi-turn tasks (search, filter, compare, order on a shopping site), this coarse signal is painful: RL delivers a single cold verdict without distinguishing which sub-step failed.

Gate one: RL gets you to the finish line, but never tells you whether each step along the way was right.

Gate Two: OPSD's Temptation and Trap

OPSD uses the same model on a parallel branch with privileged context (reference answers, retrieved skills, etc.) as a teacher, offering token-level guidance. But two critical problems emerge in multi-turn agent settings:

Observation 1: Multi-turn OPSD instability

Observation 2: Asymmetric trust in privileged guidance

not symmetrically trustworthy

Positive gap (teacher thinks the token's probability should be *higher*): the endorsement is usually credible—the student already exhibits the behavior, just underconfident.
Negative gap (teacher thinks it should be *lower*): this may indicate a genuine error—but it may also stem from flawed retrieved skills (incomplete, irrelevant, contradictory), from the teacher's inability to translate skills into reliable token-level preferences, or from accumulated multi-turn drift.

On Qwen2.5-3B, over 50% of tokens are negative-gap tokens—so indiscriminately accepting all negative feedback would suppress the student's correct behavior more than half the time.

Gate two: the teacher's word is not gospel. Praise can be fully accepted; criticism deserves caution.

Gate Three: SDAR's Solution—Each Token Decides the Window

The design philosophy: RL stays dominant (the only mechanism learning from real environment feedback, keeping optimization unbiased); OPSD is demoted to auxiliary, strictly limited.

Core design: Token-Level Gating

SDAR introduces a gating signal $g_t \in [0,1]$ on each generated token, controlling OPSD's influence. Three complementary strategies:

1. Entropy Gating: $g_t = \sigma(\beta h_t)$ — the student's entropy $$h_t$$ measures hesitation: open the window when uncertain, close it when confident. 2. Gap Gating: $g_t = \sigma(\beta \Delta_t)$ where $\Delta_t = \log \pi_T(y_t|s_t^+) - \log \pi_\theta(y_t|s_t)$ is the teacher-student log-probability gap. Positive gaps open the gate wide; negative gaps softly suppress (not hard-close) it. This realizes asymmetric trust: endorse → strong supervision; oppose → weak supervision. 3. Soft-OR Gating: $g_t = \sigma(\beta[1-(1-h_t)(1-\Delta_t)])$ — the most permissive, but empirically the weakest.

Loss function

\mathcal{L}(\theta) = \mathcal{L}_{\text{GRPO}}(\theta) + \lambda_{\text{SDAR}} \cdot \mathcal{L}_{\text{SDAR}}(\theta)

\ell_t^{\text{SDAR}} = g_t \cdot (\log \pi_\theta^+(y_t|s_t^+) - \log \pi_\theta(y_t|s_t))

Refinements worth noting:

$\pi_\theta^+$ and $\pi_\theta$ are the same model, differing only in whether privileged skills are in the input—hence "Self-Distilled."
The gate $$g_t$$ uses stop-gradient: gradients flow only through the student branch.
The GRPO loss is untouched, preserving RL's verifier-driven purity.

Experiments: Numbers Don't Lie

Benchmarks: ALFWorld (text-based household tasks), Search-QA (multi-hop search QA), WebShop (online shopping). Models: Qwen2.5-Instruct and Qwen3-Instruct, 1.7B–7B.

| Method | ALFWorld | Search-QA | WebShop-Acc | |--------|----------|-----------|-------------| | GRPO | 75.0 | - | - | | SDAR (3B) | 84.4 (+9.4%) | +7.0% | +10.2% |

The 1.7B miracle

Pure GRPO: 46.1%
Skill-GRPO (with external skills): 21.1% — worse than GRPO, indicating harmful distribution shift
RLSD: 42.2%
SDAR: 53.9%

Collapse of naive OPSD

Skill internalization vs. skill dependence

internalizes

Training dynamics

The mean teacher-student gap stays negative (the privileged teacher on average disapproves of student tokens) but steadily converges to 0.
Gating activation rate ( $$g_t > 0.5$$ ) starts strictly below 0.5—a conservative phase—then rises as the student policy improves.

Robustness to retrieval quality

random

Ablation

Deeper Reflections

SDAR touches a fundamental question: how should we treat imperfect but potentially valuable supervision signals?

A dialectic of trust: the teacher is not independent of the student—it is the same model's projection under privileged conditions. When skill retrieval is imperfect and multi-turn drift exists, the teacher's objections may carry more noise than its endorsements. SDAR's asymmetric design—amplify agreement, soften disagreement—is a mature cognitive strategy.
The boundary of auxiliary objectives: RL is the backbone; OPSD is merely the scenery outside the window. Gating ensures the auxiliary objective never overtakes the main one.
Coarse vs. fine granularity: trajectory-level RL keeps the big direction correct; token-level gated OPSD refines each step—much like human learning: know *what* to do first, then optimize *how*.
Self-distillation as autonomous learning: the "Self" means the quality of learning ultimately depends on the student's own judgment. The teacher offers perspective; whether and how much to listen, each token decides.

References

1. Lu, Z., et al. (2026). *Self-Distilled Agentic Reinforcement Learning*. arXiv:2605.15155. 2. Shridhar, M., et al. (2020). ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. arXiv:2010.03768. 3. Yao, S., et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023. 4. Jin, Q., et al. (2025). Search-R1: Training LLMs to Reason with Search. arXiv:2501.17974. 5. Shao, Z., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300. 6. Guo, D., et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948. 7. Shen, Y., et al. (2023). HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face. NeurIPS 2023. 8. Ross, S., Gordon, G., & Bagnell, D. (2011). A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS 2011.

> "The teacher on the other end of the walkie-talkie is not omniscient. But if you can tell when they're nodding versus just clearing their throat—you're already ahead of most students." > > That is what SDAR teaches us—perhaps what agent training teaches about ourselves.