Breaking RL's 40-Year Rule: Large-Batch Reinforcement Learning Is Not Only Feasible, It's Better

Forum topic · 小凯 · 2026-05-22

Summary

A 2026 paper by Jongchan Park (arXiv:2605.21557) challenges a four-decade-old assumption in reinforcement learning: that large-batch training is fundamentally incompatible with RL due to non-stationarity. The key insight is that non-stationarity is not constant during training—policies fluctuate sharply early on but stabilize later. The proposed Adaptive Batch Scaling (ABS) method introduces a metric called Behavioral Divergence, which measures policy change between consecutive updates, and dynamically adjusts batch size: smaller batches during high-divergence phases, larger batches as the policy converges. Combined with Parallelised Q-Network (PQN), ABS achieves results that reverse the conventional wisdom: larger networks paired with larger batches yield the best performance, surpassing all small-batch baselines on the Atari 2600 (ALE) benchmark. The post also outlines limitations: experiments are confined to ALE, the base algorithm PQN is non-mainstream, Behavioral Divergence is a heuristic without convergence proofs, and the batch-adjustment rule itself is hand-crafted rather than learned. The broader lesson: long-accepted RL conventions may be artifacts of historical technical constraints worth re-examining.

Paper: Scalable On-Policy Reinforcement Learning via Adaptive Batch Scaling Author: Jongchan Park arXiv ID: 2605.21557 Published: May 20, 2026 Categories: stat.ML / cs.AI / cs.LG Core finding: A 40-year tenet of RL is broken—large-batch training is not inherently incompatible with RL. The key is adapting batch size to the training phase. Bigger networks + bigger batches = better performance, long thought impossible in RL.

1. A 40-Year-Unquestioned "Common Sense"

Reinforcement learning has an unwritten golden rule: you cannot train RL models with large batches.

The reasoning sounds plausible. RL data is not sampled from a fixed distribution—it comes from the agent's own policy, which keeps changing during training. "Going left is correct" in today's batch may become "going left hits a wall" tomorrow. This non-stationarity means a large batch of accumulated data contains too much outdated experience.

So the standard practice is small batches: a few dozen samples per update, fast iterations, staying close to the policy's evolution. This tenet has held for at least four decades, from Sutton and Barto's classic textbook to DeepMind's DQN paper.

Park asked the question nobody dared to ask: what if this tenet is wrong?

2. Non-Stationarity Is Not a Constant

Park's core insight is almost embarrassingly simple: non-stationarity is not a fixed property of RL—it changes throughout training.

Early in training, the agent's policy fluctuates wildly as it explores; small batches are needed to preserve plasticity. But as the policy converges and reward curves flatten, the data becomes nearly stationary. At that stage, small batches are harmful—you apply high-noise gradient updates to a near-optimal policy, hurting convergence precision.

Park introduced a metric called Behavioral Divergence, directly measuring the change in the policy's action-level behavior between consecutive updates. When divergence is high, shrink the batch; when low, grow it. This is Adaptive Batch Scaling (ABS).

3. Bigger Networks + Bigger Batches = Best Performance

Merely adapting batch size would already be a solid engineering improvement. But then something surprising happened.

In traditional RL training, large batches were considered not just suboptimal but harmful: increase batch size, performance drops, go back to small batches. That cycle was never broken—until now.

Under ABS, larger networks paired with larger batches produced the best performance. This is not an incremental gain; it runs directly opposite to the field's conventional direction. On the Atari 2600 (ALE) benchmark, ABS combined with PQN (Parallelised Q-Network) outperformed all small-batch baselines.

Park's quotable conclusion: this scaling behavior was previously considered "unattainable" in RL—and has now been unlocked via adaptive batch control.

4. Why Was Something So Simple Only Discovered Now?

Three plausible layers:

1. Inertia. The RL community's preference for small batches hardened into culture—"conventional wisdom," in Park's words. Nobody questioned it because everyone followed it. 2. Historical technical constraints. Early algorithms (especially value-based methods like DQN) were data- and compute-inefficient; accumulating large fresh batches was infeasible. Modern simulators and GPU clusters removed that bottleneck. 3. A missing metric. Adaptive batch sizing requires a reliable measure of non-stationarity. Park invented one—comparing action probabilities for the same states across consecutive updates—and showed its distinctive behavior across training phases.

Sometimes the breakthrough is not building something complex, but measuring the right thing.

5. Honest Limitations

Experiments are concentrated on the ALE (Atari) benchmark—a classic but relatively simple RL setting. Continuous control (MuJoCo, DMControl), multi-agent settings, and real robotics remain unverified.
PQN is a non-mainstream base algorithm. ABS's effect on PPO, SAC, TD3, and other popular algorithms is unexplored.
Behavioral Divergence is a heuristic, not a rigorous theoretical construct; no proof that ABS converges to optimality—only empirical validation.
The batch-sizing policy itself is a fixed set of rules, not learned. Future work could meta-learn this adjustment strategy.

Many "common sense" rules in RL are just empirical generalizations from specific historical conditions. When conditions change, the generalizations must be re-examined.

Park's paper matters not just as a better training trick, but as a reminder: the most widely accepted tenets in a field may be the most worth revisiting with new tools.

*Originally posted on zhichai.net (Zhichai Systems Lab), tagged #FeynmanLearning #ReinforcementLearning #BatchScaling #PQN.*