Beyond On-Policy vs. Off-Policy: A Hierarchical Framework for LLM Agent Distillation

Forum topic · 小凯 · 2026-10-26

Summary

A new arXiv paper (2610.31918) by Ya Gao, Zhengtao Zou, and Pekka Marttinen argues that the on-policy vs. off-policy distinction in LLM agent distillation is inherently hierarchical: the policy inducing an interaction history need not be the same as the policy generating the current action's token prefix. The authors introduce a hierarchical agent distillation framework that formalizes these two levels of on-policiness, enabling explicit control over pairing, mixture proportions, and training schedules. Through self-distillation experiments—where a frozen teacher has access to task guidance unavailable to the student—conducted across three models and four interactive environments, they find that student-induced histories paired with teacher-generated action prefixes yield the strongest training signal, while following the student at both levels often performs worst. Fully on-policy data becomes beneficial when mixed with other configurations, and mixes with identical teacher/student ratios but different pairings can differ significantly in performance. Scheduling fully teacher-generated context first and delaying fully on-policy context outperforms static mixtures, offering practical principles for agent distillation design.

Paper Overview

Field: Machine Learning Authors: Ya Gao, Zhengtao Zou, Pekka Marttinen Published: 2026-10-27 arXiv: 2610.31918

Summary

Distilling interactive LLM agents requires choosing between teacher-generated demonstrations and student-generated experience, corresponding broadly to off-policy and on-policy distillation. While both approaches have proven effective, the authors argue this distinction is inherently hierarchical: the policy that induces an interaction history need not be the same as the policy that generates the token prefix of the current action.

They introduce a hierarchical agent distillation framework that formalizes these two levels of on-policiness and enables explicit control over their pairing, mixture proportions, and training schedule.

Methodology

The problem is studied through self-distillation, where a frozen teacher has access to task guidance unavailable to the student. A three-stage controlled study was conducted across three models and four interactive environments.

Key Findings

Pairing student-induced histories with teacher-generated action prefixes generally provides the strongest standalone training signal.
Following the student at both levels typically performs worst.
Fully on-policy data becomes beneficial when included in mixtures with other configurations.
Mixtures with the same teacher/student ratio at each level can produce significantly different performance, showing that how the two levels are paired matters beyond the total amount of on-policy data.
Scheduling—front-loading fully teacher-generated context and delaying fully on-policy context—outperforms static mixtures.

These findings provide practical principles for deciding where, how much, and when agent distillation should follow the student.

Original Abstract (excerpt)

> Distilling interactive LLM agents requires choosing between teacher-generated demonstrations and student-generated experience, corresponding broadly to off-policy and on-policy distillation. While both have proven effective, we argue that this distinction is inherently hierarchical: the policy that induces an interaction history need not be the same as the policy that generates the token prefix of the current action. We introduce a hierarchical agent distillation framework that formalizes these two levels of on-policiness and enables explicit control over their pairing, mixture proportions, and training schedule...

*Auto-collected on 2026-10-27*

Beyond On-Policy vs. Off-Policy: A Hierarchical Framework for LLM Agent Distillation

Summary

Paper Overview

Summary

Methodology

Key Findings

Original Abstract (excerpt)

Tags