Stop Random Fiddling: How KVPO Fixes Flickering in AI Video Generation

Forum topic · 小凯 · 2026-05-26

Summary

KVPO (ODE-Native GRPO) is a new reinforcement learning framework for aligning autoregressive video generation models with human intent. Traditional RL approaches inject stochastic noise for exploration, which conflicts with the deterministic ODE dynamics of modern video models, causing visual artifacts like flickering and deformation while disrupting high-level semantic coherence. KVPO solves this by shifting exploration from random noise to the model's historical KV cache: KV Semantic Exploration recombines past memory segments to generate new narrative possibilities, while a Velocity-Field Surrogate Policy based on Trajectory Velocity Energy (TVE) provides an ODE-native scoring mechanism. Experiments show KVPO improves image quality, motion coherence, and text alignment on both short single-prompt videos and long multi-segment generations. Based on the paper "KVPO: ODE-Native GRPO for Autoregressive Video Alignment via KV Semantic Exploration" (arXiv:2605.14278, May 19, 2026).

Introduction: The "Random Twitching" of Video AI

Getting AI to generate video according to human intent (alignment) is a notoriously hard problem today. Common reinforcement learning (RL) methods rely on injecting "random noise" into the model to discover new ideas — like asking a painter to scribble blindfolded and hoping for a masterpiece. The result? Grass, trees, colors, and lighting change (low-level appearance shifts), but the story's direction and motion coherence (high-level semantics) fall apart, producing twitching, deformation, and other defects.

In short: the way AI explores new frames is being done in the wrong place.

The Root Cause: Mismatched ODE Dynamics

The problem goes down to the algorithmic foundations. Many modern video models use deterministic ODE dynamics under the hood for speed — like a train speeding along fixed rails. Yet old-school RL methods still use the SDE (stochastic differential equation) playbook, forcibly injecting random noise. That's like using a bicycle repair manual to fix a high-speed train: a complete mismatch that shreds video quality.

> 💡 Tip: This is called "ODE dynamics mismatch" — applying stochastic legacy tools to today's deterministic, efficient models, causing generated frames to go off the rails.

The Breakthrough: KVPO's Approach

In May 2026, KVPO (ODE-Native GRPO) was introduced. It moves RL exploration away from meaningless random noise and into a real treasure trove — the historical KV cache:

1. KV Semantic Exploration: Instead of blindly adding noise to frames, the model reviews its own "drafts" (the historical KV cache). By randomly splicing and recombining these memory segments, it explores countless new narrative possibilities while keeping frames realistic.

2. Velocity-Field Surrogate Policy: It abandons legacy probability formulations entirely, replacing them with Trajectory Velocity Energy (TVE) — a gauge tailored to ODE models that precisely scores which narrative directions earn the highest reward.

The core velocity-matching objective is expressed as:

\mathcal{J}_{KVPO} = \mathbb{E}_{\tau} \left[ R(\tau) \cdot \log P_{\text{TVE}}(\tau | \text{ODE}) \right]

> 💡 Reading the formula: The optimization objective ( $\mathcal{J}$ ) requires the model not only to earn high reward ( $$R$$ ), but also that its generated video trajectories ( $\tau$ ) closely follow the underlying ODE's velocity field (TVE) — which is what keeps frames from collapsing.

Comparison

Results: A "Double Award Winner" for Short and Long Videos

The results are highly competitive. Whether generating short clips from a single sentence or long videos stitched from multiple prompts, models equipped with KVPO comprehensively outperform baselines in image quality, motion coherence, and prompt adherence (text alignment). It demonstrates that rather than letting AI scribble blindfolded, it's better to help it "learn new things by revisiting the old."

References

Paper: KVPO: ODE-Native GRPO for Autoregressive Video Alignment via KV Semantic Exploration
Published: May 19, 2026
arXiv ID: arXiv:2605.14278
Core problem: Solving coherence collapse in autoregressive (AR) video generation during preference alignment, caused by ODE dynamics mismatch and noise-based exploration.
Mechanism: Proposes an ODE-native GRPO framework, pioneers KV-cache-based semantic exploration, and introduces Trajectory Velocity Energy (TVE) as a surrogate policy for high-quality video alignment.