40-Year Reinforcement Learning Rule Broken: Large-Batch RL Training Is Not Only Feasible but Better

Paper: Scalable On-Policy Reinforcement Learning via Adaptive Batch Scaling

• Author: Jongchan Park
• arXiv ID: 2605.21557
• Released: May 20, 2026
• Categories: stat.ML / cs.AI / cs.LG

A four-decade-old rule of thumb in reinforcement learning says: you cannot train RL models with large batches. RL data does not come from a fixed distribution — it is generated by the agent's own policy, which keeps changing during training. What counts as the right action today may be wrong tomorrow. This non-stationarity means large batches accumulate stale experience, so the standard practice has been small batches with fast updates, from Sutton and Barto's textbook to DeepMind's DQN.

Key points

Non-stationarity is not a constant

Park's core insight is that non-stationarity varies over the course of training:
• Early training: the policy fluctuates wildly, so small batches preserve plasticity.
• Later training: the policy converges and the data becomes nearly stationary. Small batches then inject noisy gradient updates into an already near-optimal policy, hurting convergence precision.

The paper introduces Behavioral Divergence, a metric measuring the change in action-level policy behavior between consecutive updates. When divergence is high, reduce the batch; when it is low, increase it. This is Adaptive Batch Scaling (ABS).

Large network + large batch = best performance

In the ABS framework, larger networks paired with larger batches produce the best results — the opposite of conventional RL wisdom. On the Atari 2600 (ALE) benchmark, ABS combined with PQN (Parallelised Q-Network) surpasses all small-batch baselines. Park describes this scaling behavior as previously 'unattainable' in RL, now unlocked by adaptive batch control.

Why was this not discovered earlier?

The post offers three explanations: 1. Inertia: small-batch preference became cultural 'conventional wisdom' in the RL community. 2. Historical technical limits: early value-based methods like DQN were data- and compute-inefficient; gathering large fresh batches was impractical. Modern simulators and GPU clusters removed this bottleneck. 3. Missing metric: adaptive scaling requires a reliable measure of non-stationarity; Behavioral Divergence fills that gap. Sometimes the key is measuring the right thing, not building something complex.

Honest limitations

• Experiments are concentrated on the ALE (Atari) benchmark; performance on continuous control (MuJoCo, DMControl), multi-agent settings, or real robots is unverified.
• PQN is a non-mainstream base algorithm; results with PPO, SAC, or TD3 were not explored.
• Behavioral Divergence is a heuristic without theoretical convergence guarantees — only empirical validation.
• The batch-sizing rule itself is a fixed heuristic, not learned; future work could meta-learn this policy.

Conclusion

Many RL 'commonsense' rules are empirical generalizations from specific historical conditions. Park's paper is more than a better training trick — it is a reminder that widely accepted field beliefs may be the most worthwhile targets for re-examination with new tools.

Original tags: FeynmanLearning, ReinforcementLearning, BatchScaling, PQN