KVPO: ODE-Native GRPO Brings Stable, Coherent AI Video Generation

Forum topic · 小凯 · 2026-05-26

Summary

KVPO (ODE-Native GRPO) is a reinforcement learning framework for aligning autoregressive video generation models with human preferences, introduced in an arXiv paper (arXiv:2605.14278, May 19, 2026). Conventional RL methods inject random noise into diffusion models that actually follow deterministic ODE dynamics, causing a mismatch that degrades high-level semantics and produces flickering and motion distortion. KVPO solves this by moving exploration from random noise to the historical KV cache: it recombines cached memory segments to explore new narrative continuations while preserving visual realism (KV Semantic Exploration). It also replaces classical probabilistic objectives with a Velocity-Field Surrogate Policy based on Trajectory Velocity Energy (TVE), which matches the ODE velocity field. Experiments on both single-prompt short videos and multi-prompt long videos show improvements in visual quality, motion coherence, and text alignment over SDE-based exploration. This article summarizes the method and its results.

Introduction: The "Random Twitching" Problem in Video AI

Getting AI to generate video according to human intent (alignment) remains a hard problem. Most existing reinforcement learning approaches rely on adding random noise inside the model to explore new outputs — like asking a painter to scribble with eyes closed and hoping for a masterpiece. The result: the flowers and trees change color and lighting (low-level appearance changes), but story direction and motion coherence (high-level semantics) fall apart, with frequent flickering and deformation.

In short, the way AI explores new visual content is being done in the wrong place.

The Root Cause: ODE Dynamics Mismatch

Many modern video models run on deterministic ODE dynamics (ordinary differential equations) for efficiency — like a train speeding along fixed rails. But older RL methods, built around SDE (stochastic differential equations)-style random noise, keep shoving stochasticity into them. The tools simply don't fit the models, and the video output falls apart.

> 💡 Note: This is called "ODE dynamics mismatch" — applying stochastic, legacy tooling to deterministic, efficient models, causing generated video to become uncontrollable.

The Solution: KVPO

KVPO (ODE-Native GRPO) shifts the RL exploration effort away from meaningless random noise and into a real treasure trove: the historical KV cache.

1. KV Semantic Exploration: Instead of adding random noise to frames, the model revisits its own "drafts" (historical KV cache). By randomly concatenating and recombining these memory segments, it explores many new narrative continuations while keeping visuals realistic. 2. Velocity-Field Surrogate Policy: It abandons legacy probabilistic formulations in favor of Trajectory Velocity Energy (TVE), an objective tailored to ODE models that precisely scores which narrative trajectories earn the highest reward.

The core velocity-matching objective is expressed as:

\mathcal{J}_{KVPO} = \mathbb{E}_{\tau} \left[ R(\tau) \cdot \log P_{\text{TVE}}(\tau | \text{ODE}) \right]

> 💡 Interpretation: The objective $\mathcal{J}$ requires that the model not only earn high reward ( $$R$$ ), but that its generated video trajectories ( $\tau$ ) align with the underlying ODE velocity formulation (TVE), keeping the visuals from degrading.

Comparison

Results

Across both single-prompt short videos and multi-prompt long videos, models equipped with KVPO achieve across-the-board gains in visual quality, motion coherence, and text alignment compared to SDE-based exploration. The lesson: rather than letting AI scribble blindly, let it learn from what it has already generated.

References

Paper: KVPO: ODE-Native GRPO for Autoregressive Video Alignment via KV Semantic Exploration
Published: May 19, 2026
arXiv: arXiv:2605.14278
Problem addressed: Coherence collapse in autoregressive (AR) video generation alignment caused by ODE dynamics mismatch and noise-based exploration
Contribution: An ODE-native GRPO framework, a KV-cache-based semantic exploration paradigm, and a Trajectory Velocity Energy (TVE) surrogate policy for high-quality video alignment