Feynman Letter: A Gentle Introduction to the DPPO Reinforcement Learning Algorithm

Forum topic · 小凯 · 2026-05-03

Summary

This zhichai.net forum post explains the Divergence Proximal Policy Optimization (DPPO) algorithm, presented as new work from ICML 2026, using a Feynman-style analogy of training an elephant on a tightrope. The author argues that classic PPO's hard clipping mechanism—cutting off reward signals once policy updates exceed a threshold—acts like blunt scissors, producing non-smooth gradients that cause training instabilities, crashes, or 'forgetting' in large language models with vocabularies of ~100k tokens. DPPO replaces probability-ratio clipping with direct measurement of divergence (such as KL divergence) between the new and old policies, applying a smooth, continuous pull proportional to the deviation—like an electromagnetic damper rather than a hard clamp. The post claims this yields stable, industrial-grade fine-tuning for large models and draws a broader lesson: robust control in feedback systems comes from continuous, differentiable constraint forces (divergence penalties) rather than rigid if-else cutoffs.

Feynman Letter: Are You Force-Trimming AI with "Scissors," or Fitting It with a "Damper"? — A Look at the DPPO Algorithm

After reading the newly unveiled Divergence Proximal Policy Optimization (DPPO) algorithm from ICML 2026, I feel that reinforcement learning (RL) has finally shed that "crude headband constraint" in favor of a "flexible suspension system."

To help you understand why training large models so easily collapses, let's talk about "training an elephant."

1. The Status Quo: A Tightrope-Walking Elephant Tortured by "Forced Truncation"

Training AI today (e.g., using PPO for human alignment) is like teaching an elephant to walk a tightrope.

The pain point: You want the model not to take overly large steps when exploring new actions (or it falls off). How does the existing PPO algorithm handle this? It uses a pair of "scissors (Clipping)": the moment the AI's actions deviate from its old ones beyond a threshold, PPO forcibly truncates the reward signal.
The consequences: For large models—especially LLMs with vocabularies of up to 100,000 tokens—this one-size-fits-all clipping is extremely non-smooth. Gradients end up roller-coastering unpredictably, and the AI frequently "freezes up" or "forgets" during training. This is what we call "physical stalling caused by non-smooth penalties."

2. DPPO: The Animal Trainer with Built-in "Electromagnetic Damping"

No more scissors. Instead, directly measure your "physical distance" (Divergence) from the original state.

It achieves a robust leap through a shift in mathematical perspective:

Physical picture (direct estimation of divergence): Instead of looking at the crude "probability ratio," it directly computes a quantity called "divergence (such as KL divergence)." This is like attaching an extremely sophisticated electromagnetic damper cable to the elephant.
Flexible traction: When the elephant strays too far, the damper cable doesn't suddenly lock (as Clipping does)—instead, it produces a smooth pulling force proportional to the deviation. This smooth gradient backpropagation lets a hundred-billion-parameter neural network maintain a silky-smooth convergence trajectory even in a wildly vast search space.
Victory over huge vocabularies: Particularly in the LLM domain, where the vocabulary is enormous, PPO's clipping often fails. DPPO elevates large-model fine-tuning efficiency and stability to a whole new industrial-grade benchmark.

3. A Feynman-Style Judgment: Control Is "Gradient Smoothness"

So-called "stable alignment" is not achieved by drawing a few rigid red lines. It is about ensuring that, within extremely complex nonlinear feedback, every traction force the system receives is continuous and differentiable.

DPPO tells us: AI training is, at its core, an energy game between "exploration" and "conservation." When an algorithm can replace "crude truncation of probabilities" with "precise measurement of divergence," we have finally found a safety rope that lets large models roam free without ever falling off the altar.

Key takeaway: When optimizing any feedback system, stop relying on if-else logic like "block anything over 10%." Introduce your own Divergence Penalty function instead. If you can lay a layer of continuous, smooth, magnetic-field-like constraint force along your system's physical boundaries, what you tame will no longer be a trembling beast, but a world-class dancer pirouetting on the edge of a cliff.