Rethinking Reinforcement Learning: Network Depth Is the Key to Unlocking Performance

Forum topic · ✨步子哥 · 2026-01-04

Summary

A forum post discusses a reinforcement learning study arguing that, once modern architecture techniques are in place, simply increasing network depth is the decisive factor for RL performance. The proposed recipe—Contrastive RL (CRL) combined with ResNet-style residual connections, Layer Normalization, and the Swish activation—stabilizes training of very deep networks, scaling from the traditional 4 layers up to 1024 layers. Across sparse-reward locomotion (Ant, Humanoid), maze navigation, and robotic manipulation tasks, deeper networks deliver 2x to 50x performance gains over shallow baselines, outperforming baselines such as SAC, SAC+HER, TD3+HER, GCBC, and GCSL. Notably, gains are not smooth: performance jumps abruptly at depth thresholds, with an emergent-phenomenon quality. At 16 layers, a Humanoid agent shifts from falling/crawling to upright walking; at 256 layers on Humanoid U-Maze, it learns to climb over maze walls. Larger gains appear on more complex, long-horizon tasks, suggesting deep networks learn hierarchical representations for spatial structure and planning. The study is presented as the first systematic reproduction of supervised-learning-style scaling effects in RL, challenging the field's tradition of shallow networks.

Rethinking Reinforcement Learning: Depth Is the Key to Unlocking Performance

A recent study challenges a long-standing convention in reinforcement learning (RL): the reliance on shallow networks. Its central finding is that, when modern architectural techniques are applied, simply increasing network depth is the key factor that unlocks RL performance—systematically reproducing, for the first time in RL, the "scaling effects" long observed in supervised learning.

1. Technical Analysis

#### 1.1 Core Architecture Techniques

The paper's reproducible recipe is "CRL + ResNet + LayerNorm + Swish", which addresses the gradient vanishing/exploding and instability that have traditionally limited deep networks in RL:

Residual connections: Skip connections let gradients flow directly, stabilizing training and enabling networks of up to 1024 layers. Each residual block contains four "Dense → LayerNorm → Swish" units.
Layer Normalization: Normalizes across feature dimensions of a single sample rather than the batch, making it more reliable for online RL with shifting data distributions.
Swish activation: A smooth, non-monotonic activation, f(x) = x * sigmoid(x), with non-zero gradients in negative regions, mitigating neuron death and improving optimization stability.

#### 1.2 Theoretical Mechanisms

Hierarchical representations and generalization: Deep networks progressively extract representations from low-level physical features to high-level semantic concepts, which is crucial for generalization. Gains are especially large in complex maze navigation, likely because the agent learns high-level representations of spatial structure and path planning.
Emergent behavior: Performance does not improve linearly with depth; it jumps at certain thresholds. As depth grows from 4 to 16 layers, a Humanoid agent transitions from falling to upright walking; at 256 layers on Humanoid U-Maze, it learns an unconventional policy of climbing over maze walls.

2. Experimental Design and Results

Setup: Tasks span locomotion (Ant, Humanoid), maze navigation, and robotic arm manipulation, all with sparse rewards. Depth is swept from a 4-layer baseline through 8–64 layers up to 1024 layers. Baselines include SAC, SAC+HER, TD3+HER, GCBC, and GCSL.

Key results:

Deeper networks achieve 2x–50x performance improvements over the 4-layer baseline, depending on the task.
Gains correlate with task complexity:
Simple manipulation tasks: 2–5x (shallow networks suffice for small state/action spaces)
Long-horizon maze navigation: 20x+ (memory and planning benefit most from depth)
Complex Humanoid tasks: 50x+ (high degrees of freedom, complex behavior space)
Emergent thresholds: 16 layers — Humanoid switches from falling/crawling to upright walking; 256 layers — Humanoid U-Maze learns to traverse walls.

3. Broader Implications

Architecture design: RL should adopt the depth-scaling practices of supervised learning, with residual connections, LayerNorm, and smooth activations as standard stabilization tools.
Training paradigms: Combining depth with self-supervised goal-conditioned RL (CRL) opens a path toward RL scaling laws analogous to those in language and vision models.
Relation to prior knowledge: The result challenges the field's traditional assumption that shallow networks are sufficient for RL, and suggests past conclusions about RL's difficulty may partly reflect architectural limitations rather than fundamental ones.

References

Study page: https://arxiv.org/html/2503.14858v3
Related commentary: https://zhuanlan.zhihu.com/p/1985305675157497501, https://cloud.tencent.com/developer/article/2596202, https://cloud.tencent.com/developer/article/2507405