Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on a Fixed Schedule in RLVR

Forum topic · 小凯 · 2026-10-15

Summary

This paper (arXiv:2610.18853) analyzes reinforcement learning with verifiable rewards (RLVR), where costly model-generated responses motivate larger rollout batches and square-root learning-rate scaling. The authors develop a two-clock framework separating the samples needed to make progress from those consumed before training collapse. Under a local quadratic model, the square-root rule preserves per-sample first-order drift but not actual progress: larger batches take fewer, longer updates and lose more to curvature, producing a learning-rate-dependent critical batch size beyond which sample efficiency degrades. A drift-threshold hypothesis further predicts a failure clock whose sample count stays roughly constant along the square-root rule. Controlled GRPO experiments on Qwen2.5-Math-1.5B across a 16x batch range support both predictions. Ratio clipping delays failure (especially for large batches), but the progress gap persists without it; relaxing clipping trades stability for short-lived speed gains. The findings replicate on another dataset, a general checkpoint, and a 7B model. The authors use the framework to derive a practical multi-step RLVR schedule: start below the critical batch and switch to larger batches before collapse.

Overview

Paper: Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on a Fixed Schedule Field: Machine Learning Authors: Yuante Li, Junwei Quan, Kai Qiu, Kate Zhang, Yicheng Tao Published: 2026-10-13 arXiv: 2610.18853

Key Points

Reinforcement learning with verifiable rewards (RLVR) depends on expensive model-generated responses, which has led practitioners to use larger rollout batches together with square-root learning-rate scaling. While this rule was reported to approximately preserve progress per sample with one optimizer step per rollout, its behavior in practical multi-step training has been poorly understood.

Two-clock framework: The authors separate the samples needed to make progress (the progress clock) from the samples consumed before training collapse (the failure clock).
Square-root rule revisited: Under a local quadratic model, the square-root rule preserves per-sample first-order drift but *not* progress. Larger batches take fewer but longer updates and lose more to curvature, yielding a learning-rate-dependent critical batch size beyond which sample efficiency degrades.
Failure clock prediction: A drift-threshold hypothesis predicts a complementary failure clock whose sample count remains approximately invariant along the square-root scaling rule — larger batches learn slower but fail on a fixed sample schedule.
Experimental validation: Controlled GRPO experiments on Qwen2.5-Math-1.5B across a 16x batch-size range support both predictions.
Role of clipping: Ratio clipping delays failure (most notably for large batches), but the progress gap persists even with clipping. Relaxing clipping trades stability for transient speed gains.
Generality: The progress gap reappears on another dataset, a general-purpose checkpoint, and a 7B model.

Practical Takeaway

Using the two-clock framework, the authors derive a simple multi-step RLVR recipe: start training below the critical batch size, then switch to larger batches before the failure clock triggers collapse.

Source: arXiv:2610.18853