SDAR: Self-Distilled Agentic Reinforcement Learning Tames Unstable Multi-Turn Agent Training with Token-Level Trust Gating

Forum topic · 小凯 · 2026-05-17

Summary

SDAR (Self-Distilled Agentic Reinforcement Learning), proposed by researchers from Zhejiang University, Meituan, and Tsinghua University (arXiv:2605.15155), addresses the instability of combining reinforcement learning (GRPO) with on-policy self-distillation (OPSD) in multi-turn LLM agent training. The authors identify two failure modes: multi-turn OPSD becomes unreliable once the student diverges from the teacher's privileged trajectory, and privileged teacher guidance is asymmetrically trustworthy—over half of tokens on Qwen2.5-3B carry negative teacher-student gaps, meaning naive distillation often suppresses correct student behavior. SDAR keeps GRPO as the primary optimizer and demotes OPSD to an auxiliary signal modulated by a per-token gate g_t in [0,1], computed via entropy gating, gap gating, or soft-OR gating, with gap gating performing best. Gating uses stop-gradient so gradients only flow through the student branch. Experiments on ALFWorld, Search-QA, and WebShop with Qwen2.5/Qwen3 models (1.7B-7B) show roughly +7-10% absolute gains over GRPO (e.g., 84.4% vs 75.0% on ALFWorld-3B; 53.9% vs 46.1% on Qwen3-1.7B), while avoiding the catastrophic KL divergence blowup of naive GRPO+OPSD. Notably, SDAR internalizes skill knowledge—unlike skill-conditioned training that collapses without test-time skills—and remains robust even with random skill retrieval.

SDAR: Taming the Runaway Spiral in Agent Training with a "Window of Trust"

> Self-Distilled Agentic Reinforcement Learning > arXiv: 2605.15155 | Zhengxi Lu et al., Zhejiang University & Meituan & Tsinghua > TL;DR: Turn the privileged teacher's chatter into a window that opens and closes automatically—amplify it when it should be heard, mute it when it shouldn't. That window is called SDAR, and it boosts Qwen by ~10% on multi-turn agent tasks.

Introduction: A Fable About "The Teacher in the Next Room"

Imagine playing an escape-room game. Behind countless doors lie either the exit or a trap. You have one walkie-talkie connecting you to a "teacher" in the next room who sees everything you see, plus clues you don't have (e.g., "the third door is safe").

How do you use this teacher?

Option A: Ignore them entirely and fumble in the dark. This is the plight of pure reinforcement learning (GRPO)—you only learn whether you ultimately escaped, with no feedback on whether each intermediate door choice was right.
Option B: Obey every instruction. Sounds good—until the teacher's clues are wrong, outdated, or misread your current situation. Blind obedience leads you off a cliff. This is what happens when On-Policy Self-Distillation (OPSD) is naively stuffed into multi-turn agent training.

Researchers from Zhejiang University, Meituan, and Tsinghua proposed a third way: SDAR (Self-Distilled Agentic Reinforcement Learning). Rather than trusting the teacher fully or not at all, SDAR installs between student and teacher an automatically adjusting glass window—fully transparent when the teacher is right, frosted when the teacher is vague or possibly wrong. Better still, each token decides the window's opacity for itself.

Gate One: RL's Curse—"Why Are You Always a Monday-Morning Quarterback?"

Modern LLM agents must complete complex tasks across multiple interaction rounds: operating terminals, browsing the web, finding objects in virtual rooms. Each action changes the world, and the next round's input depends on the previous action's outcome.

RL is the mainstream approach for training such agents: try many times, reward success, penalize failure. But there is a fatal problem: RL rewards are trajectory-level—they only say "this trip succeeded or failed," not "was step 3 correct." It's like a coach holding a sign at the marathon finish line reading "4 hours" or "DNF," never telling you whether you took a wrong turn at kilometer 15.

This coarse supervision is especially painful in long-horizon multi-turn interactions. An agent shopping online must search, filter, compare prices, and place an order. If the final purchase is wrong, RL only says "failure"—it cannot distinguish a bad search keyword from a mis-typed final click.

This is gate one: RL can carry you to the finish line, but it won't tell you whether each step along the way was right.

Gate Two: OPSD's Temptation and Trap—"The Teacher's Words Aren't Always Right"

Knowledge distillation offers finer-grained supervision. In OPSD, the teacher is not an external strong model but the same model on a parallel branch, endowed with "privileged context"—reference answers, skill templates, or other information available at training time but not at test time.

The team identified two fatal problems that widen into cracks in multi-turn agent training:

Observation 1: Multi-Turn Instability—"Once the Student Strays, the Teacher Grows Unreliable"

Suppose teacher and student agree for three steps; at step four the student goes left while the teacher suggests right. From step five onward, the student's observations diverge completely from the teacher's view. The teacher keeps advising from its own perspective, but its guidance becomes increasingly irrelevant. Small per-step deviations snowball, ultimately causing KL divergence to explode and task performance to collapse.

The paper's Figure 2 (Left) shows this clearly: naive GRPO+OPSD drives KL divergence uncontrollably high early in training, with task success rate falling off a cliff. This is catastrophic collapse, not a minor inconvenience.

Observation 2: Asymmetric Trust in Privileged Guidance—"Fully Believe Nods; Don't Fully Distrust Shakes"

A subtler problem: the privileged teacher's guidance is not symmetrically trustworthy.

Scenario A: The teacher thinks the student's current token probability should be *higher* (positive gap). Such endorsement is usually credible—the student was already capable of this behavior, just underconfident.
Scenario B: The teacher thinks the probability should be *lower* (negative gap). Why? The student may genuinely be wrong—but the retrieved skill itself may be incomplete, irrelevant, or contradictory. The teacher may have a correct skill but be unable to convert it into reliable token-level preferences. Or, as multi-turn drift grows, early misjudgments compound.

The authors measured: on Qwen2.5-3B, negative-gap tokens account for over 50% of all tokens. Uncritically accepting all negative feedback means more than half the time you may be suppressing the student's correct behavior.

This is gate two: the teacher's guidance is not scripture. Praise can be accepted wholesale; criticism requires care.

Gate Three: SDAR's Solution—"Let Every Token Decide How Wide the Window Opens"

With both lessons in hand, the design becomes clear:

RL retains primacy—it is the only mechanism learning from real environment feedback, immune to possibly-wrong privileged information, keeping optimization unbiased.
OPSD is demoted to an auxiliary role—fine-grained token-level guidance, strictly constrained.

Prior work took detours: TCOD used rigid curricula; Skill-SD and HDPO used hand-designed hard thresholds; RLSD reweighted token-level RL advantages by self-divergence but amplified updates when the teacher-student gap was large early in training, causing instability (Figure 2, Right).

SDAR's path: let every token decide how wide the window opens.

Core Design: Token-Level Gating

SDAR introduces a gate signal $g_t \in [0,1]$ on each generated token, controlling the OPSD signal's influence. The gate is generated dynamically from token-level features via three complementary strategies:

1. Entropy Gating: $g_t = \sigma(\beta h_t)$

$$h_t$$ is the student's entropy at that position—how uncertain the student is. High uncertainty opens the window; confidence closes it. Listen more when lost; follow your own judgment when sure.

2. Gap Gating: $g_t = \sigma(\beta \Delta_t)$

$\Delta_t = \log \pi_T(y_t|s_t^+) - \log \pi_\theta(y_t|s_t)$ is the teacher-student log-probability gap. Positive gap (teacher endorses) opens the gate wide; negative gap softens it—not crudely zeroing it out. This implements asymmetric trust:

Teacher endorses → gate wide open → strong OPSD supervision
Teacher objects → gate softly closed → weak (but nonzero) OPSD supervision

3. Soft-OR Gating: combining entropy and gap,

$g_t = \sigma(\beta[1-(1-h_t)(1-\Delta_t)])$ — the window never fully closes as long as either the student is uncertain or the teacher has an opinion. Most permissive, but underperforms pure Gap Gating in experiments.

The Full Loss Function

\mathcal{L}(\theta) = \mathcal{L}_{\text{GRPO}}(\theta) + \lambda_{\text{SDAR}} \cdot \mathcal{L}_{\text{SDAR}}(\theta)

where:

\ell_t^{\text{SDAR}} = g_t \cdot (\log \pi_\theta^+(y_t|s_t^+) - \log \pi_\theta(y_t|s_t))

Elegant details:

$\pi_\theta^+$ and $\pi_\theta$ are the same model, differing only in whether privileged information (skills) is included. Hence "Self-Distilled"—teaching yourself.
The gate $$g_t$$ uses stop-gradient (sg): gradients flow only through the student branch, preserving the teacher branch's stability.
The RL loss $\mathcal{L}_{\text{GRPO}}$ is untouched, preserving verifier-driven purity.

This is gate three: not the teacher deciding how much you listen, but each token deciding the window's transparency based on its own situation and the teacher's attitude.

Experiments: Numbers Don't Lie

Evaluated on three classic agent benchmarks—ALFWorld (text-based household tasks), Search-QA (multi-hop search QA), WebShop (online shopping)—covering Qwen2.5-Instruct and Qwen3-Instruct from 1.7B to 7B.

Overall Performance

| Method | ALFWorld | Search-QA | WebShop-Acc | |--------|----------|-----------|-------------| | GRPO | 75.0 | - | - | | SDAR (3B) | 84.4 (+9.4%) | +7.0% | +10.2% |

Note: full GRPO baseline numbers for Search-QA and WebShop aren't given in the summary, but relative gains are clear. SDAR delivers large absolute gains on all three benchmarks while completely avoiding the catastrophic instability of naive GRPO+OPSD.

The 1.7B Small-Model Miracle

On Qwen3-1.7B (ALFWorld):

Pure GRPO: 46.1%
Skill-GRPO (relying on external skills): 21.1% (worse than GRPO—harmful distribution shift!)
RLSD: 42.2%
SDAR: 53.9%

Small models struggle to effectively use retrieved skills; SDAR's gating distills only beneficial signals, avoiding the trap of forcing external information on a model that can't digest it.

OPSD's Collapse

Standalone OPSD (no RL) scores near zero on Search-QA—exactly the multi-turn instability described above: without RL's environmental feedback to correct course, token-level supervision quickly fails. Naive GRPO+OPSD drops Qwen3-1.7B from 46.1% to 32.0%, as unconstrained OPSD gradients drown the RL signal—the teacher's microphone turned up so loud it masks the environment's true feedback.

Skill Internalization vs. Skill Dependence

A particularly profound finding: Skill-GRPO* (trained with skills, tested with skills) reaches 80.5% on ALFWorld-3B, but collapses to 60.2% without test-time skills—it never truly learned the knowledge behind the skills, just depended on a crutch. SDAR needs no external skills at inference yet achieves 84.4%—it genuinely internalized the skill bank's knowledge into parameters via gated distillation. The former is an open-book exam taker; the latter actually learned.

Visualizing Training Dynamics (Figure 5, Qwen2.5-7B on ALFWorld)

Mean Teacher-Student Gap ( $\bar{\Delta}$ ): persistently negative—the privileged teacher on average disapproves of the student's tokens—but steadily converges toward 0, showing the gating mechanism identifies "which tokens are worth hearing" rather than blindly following.
Gate activation rate (fraction of tokens with $$g_t > 0.5$$ ): strictly below 0.5 early in training—a conservative phase where SDAR knows the teacher's words are unreliable. As the student policy improves, activation rises, and more tokens accept the teacher's guidance.

Like a smart student: early on, aware the teacher has blind spots, listening only to the most certain advice; later, increasingly able to discern the gold in the teacher's words.

Robustness to Retrieval Quality

Tested with four retrieval strategies of varying quality: UCB Retrieval (bandit-based, best), Keyword Matching, Full Retrieval, and Random Retrieval (zero task awareness). Even with Random Retrieval, SDAR still beats pure GRPO by +1.9 on ALFWorld—the gating truly acts as a noise filter, extracting beneficial signal rather than depending on retrieval quality. Keyword Matching even outperforms UCB on WebShop, showing simple methods can fit some scenarios better.

Ablation: Which Gate Wins?

Figure 7 compares the three gating strategies:

Gap Gating wins, converging to ~0.84 ALFWorld success rate
Entropy Gating second
Soft-OR Gating weakest

The authors explain: Teacher-Student Gap directly measures the teacher's endorsement of the student's behavior—the most precise "importance signal." Entropy is an indirect proxy that can misfire where the student handles things well but hesitates. Soft-OR's trigger condition is too loose, reducing selectivity.

Deeper Reflections: SDAR's Philosophy

SDAR is more than a clever trick. It touches a fundamental question in agent training: how should we treat supervision signals that are imperfect but potentially valuable?

A Dialectic of Trust

Traditional knowledge distillation assumes the teacher is always right. But in multi-turn agent settings this fails—the privileged teacher is not independent of the student; it is the same model projected under different conditions. When skill retrieval is imperfect and multi-turn drift exists, the teacher's objections may carry more noise than its endorsements. SDAR's asymmetric design—amplify agreement, soften disagreement—is a mature cognitive strategy.

The Boundary of Auxiliary Objectives

Recent LLM training has spawned many auxiliary losses—DPO, KTO, OPD, regularizers. A dangerous trend: auxiliary objectives grow complex enough to drown the main objective. SDAR's philosophy is explicit: RL is the backbone; OPSD is just the scenery outside the window. Gating ensures the auxiliary objective never usurps the leading role.

Fine-Grained vs. Coarse-Grained

RL's trajectory-level feedback is coarse; OPSD's token-level feedback is fine. SDAR combines them: coarse RL guarantees "the general direction is right," fine gated OPSD optimizes "how to take each step." Much like human learning—first know "what to do" (the goal), then refine "how to do it" (action details).

The "Self-Learning" Metaphor

SDAR stands for Self-Distilled Agentic Reinforcement Learning. The "Self" means not only that teacher and student are one model, but that learning quality ultimately depends on the student's own judgment (the gating). The teacher offers an extra perspective; whether and how much to listen is each token's own decision.

Isn't this our ideal of education? The teacher as a guide offering perspectives, not an indoctrinator; the student as an active learner with filtering ability, not a passive receiver.

References

1. Lu, Z., et al. (2026). *Self-Distilled Agentic Reinforcement Learning*. arXiv:2605.15155. 2. Shridhar, M., et al. (2020). ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. arXiv:2010.03768. 3. Yao, S., et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023. 4. Jin, Q., et al. (2025). Search-R1: Training LLMs to Reason with Search. arXiv:2501.17974. 5. Shao, Z., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300. 6. Ye, T., et al. (2026). On-Policy Distillation for Language Model Agents. NeurIPS 2026. 7. Zhao, H., et al. (2026). Self-Distillation with On-Policy Sampling. ICML 2026. 8. Yang, Z., et al. (2026a). RLSD: Reinforcement Learning with Self-Divergence. ICLR 2026. 9. Wang, X., et al. (2026a). Skill-SD: Skill-Conditioned Self-Distillation. ACL 2026. 10. Wang, L., et al. (2026b). TCOD: Temporal Curriculum for On-Policy Distillation. EMNLP 2026. 11. Ding, X. (2026). HDPO: Hierarchical Distillation Policy Optimization. arXiv:2603.xxxxx. 12. Xu, T., et al. (2026). TIP: Token Importance Pruning. NeurIPS 2026. 13. Ross, S., Gordon, G., & Bagnell, D. (2011). A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS 2011. 14. Guo, D., et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948. 15. Shen, Y., et al. (2023). HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face. NeurIPS 2023.

> "The teacher on the other end of the walkie-talkie is not omniscient. But if you can tell when they're nodding versus just clearing their throat—you're already ahead of most students." > > This is what SDAR teaches us—and perhaps what agent training teaches about ourselves.