Overview
Paper: JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion arXiv: 2608.03974 Team: JD Joy Future Academy Category: cs.CV
A 16B-parameter autoregressive diffusion model that edits 720p video at ~30 FPS on a single NVIDIA B200 GPU, without knowing the video length in advance.
Why real-time video editing is hard
Two worlds of video editing have remained disconnected:
- Offline editors (VACE, OpenVE-Edit, Bernini-R) read the entire clip with bidirectional attention and produce high-quality results, but cannot operate on live streams.
- Streaming editors (StreamDiffusionV2, SANA-Streaming, LiveEdit, XMax-X2.0) process frames causally and run in real time, but their OpenVE-Bench scores typically fall between 1.2 and 2.6 (out of 5), well below offline systems above 3.6.
- An MLLM that converts a natural-language instruction and the first frame into conditioning.
- A causal VAE with 8×24×24 spatio-temporal compression.
- An MM-DiT backbone.
- vs LiveEdit: 90%
- vs SANA-Streaming: 87%
- vs StreamDiffusionV2: 87%
- vs XMax-X2.0: 81%
- vs Bernini-R (offline): 48% vs 44% (statistical tie)
- vs Kling-3.0 Omni / Seedance 2.0 (closed commercial): 56%
- Long-video drift is quantified for the first time on a 1-minute benchmark; unoptimized streaming methods lose roughly half their quality from 10-second to 1-minute clips.
- SA-DMD shifts inference-time burden to training time, a pattern applicable to real-time speech translation and face reconstruction.
- System-level co-design (VAE + DiT + cache + quantization) matters as much as the core algorithm.
- Open challenge: 30 FPS still requires 16B parameters and a B200. Consumer hardware support is the next milestone.
- Yicheng Xiao, Wenxun Dai, et al. "JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion." arXiv:2608.03974, 2026.
- JD Joy Future Academy: https://research.joyai.com/
The paper asks: can editing quality match offline models while latency matches live streams?
The train-inference mismatch trap
Simply removing future-frame attention (causalizing an offline model) causes long-term drift. Models are trained with teacher forcing on clean ground-truth history, but at inference they condition on their own imperfect outputs. Errors compound across seconds and minutes, causing color shift, identity drift, and loss of fine background detail. Video editing is harder than generation because edits must remain temporally consistent with the original source video while faithfully executing the edit instruction.
Three technical contributions
1. Chunk-wise autoregressive generation
The architecture combines:
The video is split into chunks along the time axis. Each chunk uses bidirectional attention internally, but cross-chunk attention is strictly causal. A sliding window keeps only the recent history plus the first chunk as a global anchor, so the per-step compute stays bounded regardless of stream length.
Resampling Forcing: instead of feeding clean ground-truth history during training, the model runs a single denoising step on each historical chunk to produce its own noisy history (no gradient), then trains against this realistic, flawed context.
2. SA-DMD (Source-Anchored Distribution Matching Distillation)
Standard DMD compresses many denoising steps into few, but for video editing the student increasingly drifts from the source. SA-DMD conditions the teacher on the time-aligned source video frames during distillation training only, decoupling text-instruction guidance strength from source-fidelity guidance strength via classifier-free guidance.
The key insight: source anchoring is applied to the training objective, not to deployment. Inference uses a single forward pass with no extra source reference. Distillation reduces denoising to two steps.
3. LHAD (Long-Horizon Autoregressive Distillation)
Short training clips never expose the model to the deep error states that appear after many seconds or minutes of rollout. LHAD splits a long chunked sequence into short segments, computes SA-DMD gradients per segment with detached computation graphs (to avoid OOM), and accumulates them for one parameter update. A dynamic mirror looping scheme flips source video forward and backward to extend training sequences without visible looping artifacts.
Experimental results
Short-video editing (OpenVE-Bench, Gemini judge, score out of 5)
| Method | Params | Resolution | Overall | |---|---|---|---| | StreamDiffusionV2 | 1.3B | 480×832 | 1.23 | | SANA-Streaming | 2B | 704×1280 | 2.62 | | LiveEdit | 1.3B | 480×832 | 2.00 | | XMax-X2.0 | — | 832×1440 | 1.87 | | Bernini-R (offline) | 27B | 480×848 | 3.72 | | JoyAI-Video-Edit | 16B | 720×1280 | 3.60 |
JoyAI ranks first among streaming methods on four of five task categories and matches or exceeds some offline systems (Bernini-R overall 3.72 vs 3.60, tied on local edit at 4.47).
Long-video editing (LongV2VBench, 229 tasks, 1-minute videos, paper-introduced)
| Method | Resolution | Throughput | Overall | |---|---|---|---| | StreamDiffusionV2 | 480×832 | 18.07 FPS | 1.21 | | SANA-Streaming | 704×1280 | 14.51 FPS | 1.64 | | LiveEdit | 480×832 | 15.45 FPS | 1.23 | | XMax-X2.0 | 832×1440 | 20.90 FPS | 1.71 | | JoyAI-Video-Edit | 720×1280 | 30.19 FPS | 3.30 |
JoyAI nearly doubles the best streaming baseline and is 44.4% faster than XMax-X2.0 despite similar resolution.
Ablation (LongV2VBench)
| SA-DMD | LHAD | Overall | |---|---|---| | ✗ | ✗ | 2.81 | | ✓ | ✗ | 3.23 | | | ✓ | 3.06 | | ✓ | ✓ | 3.30 |
SA-DMD mainly suppresses visual drift (global style 3.61→4.24, local edit 3.43→4.00). LHAD mainly stabilizes late stages (background swap 2.45→2.60, local removal 2.58→2.70). Combined, the two complement each other.
Human preference (pairwise, blind)
Deployment performance
Per 8-frame chunk on one B200:
| Component | Time | |---|---| | VAE encode | 22 ms | | DiT denoise (2 steps) | 185 ms | | VAE decode | 19 ms | | Request→response | 226 ms | | + KV cache build + pseudo encode | 266 ms | | Effective frame rate | ≈30.1 FPS |
Engineering levers: FP8 quantization, compiled VAE with auto-tuning, pipelined host/GPU overlap, warm-up, persistent compile artifacts, and a reserved memory pool.
Comparison on 81-frame inputs, batch size 1:
| Method | Resolution | Total latency | FPS | VAE FPS | |---|---|---|---|---| | StreamDiffusionV2 | 480×832 | 4.48s | 18.07 | 37.06 | | LiveEdit | 480×832 | 5.24s | 15.45 | 37.26 | | SANA-Streaming | 704×1280 | 5.58s | 14.51 | 27.12 | | JoyAI | 720×1280 | 2.68s | 30.19 | 200.00 |
JoyAI delivers lowest latency and highest throughput at the highest resolution, indicating gains come from end-to-end co-design rather than a single algorithmic trick.
Training data: two routes for paired-data scarcity
1. Key-frame-guided edit propagation: pick a representative frame, edit it with an image-editing model, then propagate the edit to all frames with an image+video→video model that preserves motion and untouched regions. 2. Latent-shared image-to-video generation: generate video from both the original and edited images, sharing latents in early denoising (consistent motion/composition) and diverging in late denoising (introducing the edit).
Quality control filters by visual quality, edit correctness, content change magnitude, and temporal consistency; a multimodal LLM verifies source-vs-edited differences and refines the text instructions.