ASTEROID: Asymmetric On-Policy Distillation Learns In-Context Exploration

Forum topic · 小凯 · 2026-10-23

Summary

Researchers from the University of Washington and collaborators introduce ASTEROID (arXiv:2610.30105), a student-teacher supervised learning framework that teaches agents to explore at test time without reinforcement learning. A privileged expert with full state access never needs to explore, but a history-conditioned student can learn exploration when trained with on-policy contexts: the student first collects context under partial observability, and the expert then provides action labels conditioned on that context. Distilling expert actions onto the student's own on-policy history teaches it to search when uncertain and exploit once hidden states are revealed, behavior resembling Bayesian posterior sampling. Across 11 environments including discrete exploration benchmarks, robotic manipulation, navigation, and Procgen, ASTEROID achieves roughly 2x higher returns under a fixed exploration budget, uses about 100x fewer interactions than RL baselines, and outperforms frontier models on exploration. Simulated policies also transfer to visual-free real-world robot manipulation, improving success rates by 40% over standard distillation.

Paper Overview

Field: ML Authors: Sriyash Poddar, Yanda Bao, Jacob Krantz, Matthew Chang, Xavier Puig, Roozbeh Mottaghi, Natasha Jaques, Abhishek Gupta Published: 2026-10-21 arXiv: 2610.30105

Introduction

Sequential decision-making under partial observability is difficult because agents must explore to infer hidden, task-relevant information before acting effectively. Learning such information-gathering behaviors tabula rasa via online reinforcement learning is sample-inefficient, whereas offline imitation averages expert actions across hidden states and fails to learn exploration.

Method

ASTEROID is a simple and effective student-teacher supervised learning framework that can learn test-time exploration. Although a privileged expert never needs to explore, the authors demonstrate that a history-conditioned student can learn to do so when trained with on-policy contexts:

The student first collects on-policy context under partial observability.
The privileged expert then provides action labels conditioned on that context.
By distilling expert actions directly onto the student's own on-policy history, the student learns to systematically resolve partial observability—searching when uncertain and exploiting once the hidden state is revealed.

This yields efficient test-time exploration (akin to Bayesian posterior sampling) without the impractical training-data costs of reinforcement learning.

Results

Across 11 diverse environments—including discrete exploration benchmarks, robotic manipulation, navigation, and Procgen—ASTEROID:

Achieves approximately 2x higher returns under a fixed exploration budget
Uses roughly 100x fewer interactions than RL baselines
Outperforms frontier models on exploration capability

Additionally, policies trained in simulation transfer to real-world, visual-free robotic manipulation, exhibiting efficient exploration and improving success rates by 40% compared to standard distillation.

Conclusion

ASTEROID offers a simple yet surprisingly effective framework for learning efficient test-time exploration through asymmetric on-policy distillation.

---

*Auto-collected on 2026-10-23*