[论文] Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedu...

论文概要

研究领域: ML
作者: Yuante Li, Junwei Quan, Kai Qiu, Kate Zhang, Yicheng Tao
发布时间: 2026-10-13
arXiv: 2610.18853

中文摘要

可验证奖励强化学习（RLVR）依赖昂贵的模型生成响应，这促使人们使用更大的 rollout 批次和平方根学习率缩放。该规则被报道在每次 rollout 一步优化时近似保持每样本进展，但其在实际多步训练中的理解仍不充分。我们发展了一个「双时钟「框架，将实现进展所需的样本与崩溃前消耗的样本分开。在局部二次模型下，平方根规则保持每样本一阶漂移但不保持进展：更大的批次进行更少、更长的更新，因曲率损失更多，产生依赖于学习率的临界批次，超过该批次后样本效率下降。漂移阈值假说预测了一个互补的失败时钟，其样本计数沿平方根规则保持近似不变。在 Qwen2.5-Math-1.5B 上跨 16 倍批次范围的受控 GRPO 实验支持这两个预测。比率裁剪延迟失败（对大批次尤其明显），但没有它进展差距依然存在。放宽裁剪以稳定性换取短暂的速度增益。进展差距在另一个数据集、通用检查点和 7B 模型上重新出现。最后，我们使用双时钟框架推导出一个简单的多步 RLVR 方案：从低于临界批次开始，在失败前切换到更大批次。

原文摘要

Reinforcement learning with verifiable rewards (RLVR) relies on costly model-generated responses, motivating larger rollout batches and square-root learning-rate scaling. Reported to approximately preserve progress per sample with one optimizer step per rollout, this rule is less understood in practical multi-step training. We develop a two-clock framework that separates the samples needed to make progress from those consumed before collapse. Under a local quadratic model, the square-root rule preserves first-order drift per sample but not progress: larger batches take fewer, longer updates and lose more to curvature, yielding a learning-rate-dependent critical batch beyond which sample efficiency degrades. A drift-threshold hypothesis predicts a complementary failure clock whose sample co...

自动采集于 2026-10-15

#论文 #arXiv #ML #小凯