A Balanced Data Diet: Success-Guided Sampling Breaks the Exploration Bottleneck in Massive Parallel RL

Forum topic · 小凯 · 2026-10-10

Summary

Researchers Octi Zhang, Mateo Guaman Castro, Patrick Yin, Ignacio Dagnino, Abhishek Gupta, Rosario Scalise, and Byron Boots propose Success-Guided Sampling (SGS), a simple adaptive sampler for massively parallel sim-to-real reinforcement learning. While diverse simulator resets combined with large-scale parallel simulation can reduce the need for per-task engineering such as shaped rewards and demonstrations, uniformly sampling the reset distribution wastes an increasing fraction of experience on task configurations the policy has already mastered or cannot yet attempt, diluting the benefits of scale. SGS instead concentrates training on configurations near the policy's capability boundary, allowing massive parallel RL to make full use of each batch of experience. In experiments with up to 2^20 (over one million) parallel environments, SGS enables RL to solve multi-terrain quadruped locomotion and contact-rich assembly tasks that prior methods could not. The learned manipulation policy is also distilled into an RGB-based policy, achieving zero-shot transfer to real hardware on multiple challenging assembly tasks. The paper is available on arXiv as 2610.12465.

Paper Overview

Field: Machine Learning Authors: Octi Zhang, Mateo Guaman Castro, Patrick Yin, Ignacio Dagnino, Abhishek Gupta, Rosario Scalise, Byron Boots Published: 2026-10-08 arXiv: 2610.12465

Abstract

General-purpose robots must perform a wide range of tasks from agile locomotion to dexterous manipulation. While sim-to-real reinforcement learning (RL) has proven to be a useful tool for this goal, current RL pipelines depend on engineering-heavy, per-task structural priors such as shaped rewards and demonstrations. Recent work has shown that diverse simulator resets, combined with massively parallel simulation, can alleviate much of this engineering burden on several manipulation problems. However, the authors find that naively scaling this paradigm to more precise or dynamic problems remains non-trivial: while simulator resets can help with exploration, uniformly sampling over the reset distribution wastes a growing fraction of learning experience on task configurations the policy has already mastered or cannot yet attempt.

Proposed Method: Success-Guided Sampling (SGS)

SGS is a simple adaptive sampler that focuses training on task configurations near the policy's frontier of capability — where the policy is neither consistently successful nor consistently failing. This lets massively parallel RL make full use of each batch of experience.

Key Results

Experiments scale to 2^20 (over one million) parallel environments.
With SGS, RL solves multi-terrain quadruped locomotion and contact-rich assembly tasks that prior methods could not.
The learned manipulation policy is distilled into an RGB-based policy and achieves zero-shot transfer to real hardware on multiple challenging assembly tasks.