DPPO: Replacing PPO's Clipping with Smooth Divergence Penalties

Forum topic · 小凯 · 2026-05-03

Summary

A Chinese tech forum post explains the Divergence Proximal Policy Optimization (DPPO) algorithm, presented as an ICML 2026 paper, using a Feynman-style analogy of training an elephant on a tightrope. The author argues that PPO's clipping mechanism acts like a blunt pair of scissors: once a policy's actions deviate beyond a threshold, reward signals are abruptly truncated. For large language models with vocabularies of around 100,000 tokens, this non-smooth penalty causes erratic gradients, training collapse, or 'amnesia.' DPPO instead directly measures a divergence quantity (such as KL divergence) between the current and reference policies, applying a smooth, proportional restoring force rather than a hard cutoff. The post claims this yields stable, industry-grade fine-tuning for large models and generalizes as a design lesson: replace binary if-else thresholds in feedback systems with continuous, differentiable divergence penalty functions for smoother, more robust control.

DPPO: Would You Trim AI with Scissors, or Fit It with a Damper?

After reading the newly released Divergence Proximal Policy Optimization (DPPO) algorithm (ICML 2026), the author feels reinforcement learning has finally shed its "crude headband" in favor of "flexible suspension."

To explain why training large models collapses so easily, the post uses the analogy of training an elephant to walk a tightrope.

1. The Status Quo: An Elephant Tortured by Forced Truncation

Training AI with PPO (e.g., for human alignment) is like teaching an elephant to walk a tightrope:

Pain point: You want the model not to take overly large steps when exploring new actions (it might fall). How does PPO handle this? With a pair of "scissors (Clipping)": as soon as the AI's actions deviate from the old policy beyond a threshold, PPO forcibly truncates the reward signal.
Consequence: For large models—especially LLMs with vocabularies of up to 100,000 tokens—this one-size-fits-all clipping is extremely non-smooth. Gradients rollercoaster, and the AI frequently "freezes" or "forgets" during training. The author calls this "physical stall caused by non-smooth penalties."

2. DPPO: The Trainer with Built-In Electromagnetic Damping

DPPO's logic is elegant: drop the scissors; directly measure the "physical distance (Divergence)" from the original state.

It achieves robustness through a mathematical shift in perspective:

Physical picture (direct divergence estimation): Instead of looking at the crude "probability ratio," it directly computes a divergence quantity (such as KL divergence). This is like attaching a highly advanced electromagnetic damping cable to the elephant.
Soft traction: When the elephant drifts too far, the damping line doesn't suddenly lock (as clipping does)—it exerts a smooth force proportional to the deviation. This smooth gradient propagation lets hundred-billion-parameter neural networks maintain silky convergence even in a wildly vast search space.
Victory for huge vocabularies: In the LLM domain especially, PPO's clipping often fails due to the enormous vocabulary. DPPO reportedly raises fine-tuning efficiency and stability for large models to a new industrial-grade benchmark.

3. The Feynman-Style Judgment: Control Is "Gradient Smoothness"

"Stable alignment" isn't achieved by drawing a few rigid red lines, but by ensuring that in highly complex nonlinear feedback, every force the system receives is continuous and differentiable.

DPPO suggests that AI training is, at its core, an energy game between "exploration" and "conservation." When an algorithm can replace crude probability truncation with precise divergence measurement, we find a safety rope that lets large models explore freely without falling off the pedestal.

Takeaway: When optimizing any feedback system, stop relying on blunt if-else logic like "block if it exceeds 10%." Introduce your own "Divergence Penalty function" instead. If you can lay down a continuous, smooth constraint—like a magnetic field—along your system's physical boundaries, what you tame will no longer be a trembling beast, but a dancer capable of pirouettes at the cliff's edge.

*Note: This is an English translation of a Chinese forum post; the claims about DPPO reflect the original author's reading of the work.*