Lip Forcing: Few-Step Autoregressive Diffusion for Real-Time Video Lip Synchronization

Forum topic · 小凯 · 2026-06-11

Summary

Lip Forcing is presented as the first autoregressive diffusion method for video-to-video (V2V) lip synchronization. The approach distills a 14B-parameter audio-conditioned bidirectional video diffusion teacher into causal student models that generate each chunk with only two denoising steps at inference, requiring no inference-time classifier-free guidance (CFG). A lip-sync-specific teacher-trajectory analysis reveals a fidelity-sync tradeoff: no-CFG predictions favor reference fidelity, while CFG-guided predictions favor synchronization within a mid-trajectory band. Performance results show the 1.3B student model achieving 31 FPS real-time lip synchronization, running 17.6x faster than a bidirectional model of comparable size, while the 14B student is 39.8x faster than its teacher. First-frame latency is sub-millisecond, making the method practical for real-time streaming applications such as virtual avatars and live dubbing. The paper was released on arXiv (2606.11180) on June 9, 2026.

Paper Overview

Field: Computer Vision (CV) Authors: Paul Hyunbin Cho, Jinhyuk Jang, SeokYoung Lee, Joungbin Lee, Siyoon Jin, Heeseong Shin, Jung Yi, Yunjin Park, Chulmin Park, Seungryong Kim Published: 2026-06-09 arXiv: 2606.11180

Abstract

Diffusion-based lip synchronization models achieve strong visual quality and audio-visual alignment, but full-sequence bidirectional attention and many denoising steps make them impractical for real-time inference. The authors present Lip Forcing, to their knowledge the first autoregressive diffusion method for video-to-video (V2V) lip synchronization, which distills a 14B audio-conditioned bidirectional video diffusion teacher into causal students.

At inference, the students generate each chunk in only two denoising steps without inference-time CFG, enabling real-time lip synchronization.

A lip-sync-specific teacher-trajectory analysis reveals a CFG fidelity-sync tradeoff: no-CFG predictions favor reference fidelity, whereas CFG-guided predictions favor synchronization within a mid-trajectory band.

Key Results

First autoregressive diffusion method for V2V lip synchronization
Teacher: 14B audio-conditioned bidirectional video diffusion model, distilled into causal students
Only 2 denoising steps per chunk at inference, no inference-time CFG
1.3B student: 31 FPS, 17.6x faster than same-scale bidirectional models
14B student: 39.8x faster than its teacher
Sub-millisecond first-frame latency, enabling real-time streaming