JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion — Deep Dive
> Paper: JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion > arXiv: 2608.03974 | Team: JD Joy Future Academy | Category: cs.CV > In one line: 16B parameters, 720p @ 30 FPS, real-time editing with no prior knowledge of video length — JD turns video editing from "post-production" into "live-synced."
Why must video editing happen "after filming"?
Imagine you are live-streaming, kicking a football in a desert. Chat floods: "Change it to a rainy stadium!" — and a second later the scene really becomes a rainy stadium: raindrops, puddles, soaked jersey, all rendered in real time. Then: "Put on the Iron Man suit!" — metallic armor appears instantly, with reflections, joint articulation, and environmental interaction. This is not post-production; it happens live. That is the goal JoyAI-Video-Edit achieves.
Two disconnected worlds of video editing
World 1: Offline editing (VACE, OpenVE-Edit, Bernini-R). The model reads the entire video with bidirectional attention and iterative denoising. Excellent quality, but you must wait until the video is finished — impossible for live streaming.
World 2: Streaming/real-time editing (StreamDiffusionV2, SANA-Streaming, LiveEdit, XMax-X2.0). Causal, frame-by-frame generation. Real-time, but with heavy quality loss: streaming methods score only 1.2–2.6 on OpenVE-Bench (out of 5), while strong offline models exceed 3.6. Blur, color drift, and poor instruction following are common.
The core question: can we achieve offline-level editing quality and live-level responsiveness?
Why naive causalization fails
Simply making an offline model causal creates a train-inference mismatch. During training, teacher forcing gives the model clean ground-truth history; at inference it must condition on its own imperfect generations. Errors compound like a game of telephone — long-term drift in color, identity, and background detail. Video editing is even harder than pure generation: the model must stay aligned with the source video (preserve unedited content) while reliably executing edit instructions — a dynamic balance.
The three key techniques
1. Chunk-wise autoregression
- MLLM: encodes the natural-language instruction plus the first frame into conditioning.
- Causal VAE: compresses video at 8×24×24 (8 frames → 1 latent frame).
- MM-DiT: the multimodal diffusion Transformer backbone.
- Videos are split into chunks with bidirectional attention inside each chunk and causal attention across chunks.
- A sliding window of recent chunks + the first chunk as a global anchor keeps context bounded regardless of stream length — like a simultaneous interpreter who remembers the opening tone plus the last few minutes.
- Resampling Forcing: each historical chunk is first run through one denoising step by the model itself (no gradient), so training uses imperfect, self-generated history — matching inference conditions.
- 3.60 overall, nearly a full point above the best prior streaming method (2.62); first place in four of five task categories; tied for best on local edits (4.47); within 0.12 of the 27B offline Bernini-R.
- Nearly double the score of the runner-up, and the highest throughput (44.4% faster than XMax-X2.0 despite lower resolution). Competing streaming methods degrade sharply on longer videos, confirming the severity of long-term drift.
- vs LiveEdit: 90% · vs SANA-Streaming: 87% · vs StreamDiffusionV2: 87% · vs XMax-X2.0: 81%
- vs offline Bernini-R: 48% vs 44% — a statistical tie with a top offline system
- vs commercial closed models Kling-3.0 Omni / Seedance 2.0: 56% — slightly ahead
- Yicheng Xiao, Wenxun Dai, et al. "JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion." arXiv:2608.03974, 2026.
- JD Joy Future Academy: https://research.joyai.com/
- Project page: to be added if publicly released.
2. SA-DMD (Source-Anchored Distribution Matching Distillation)
Real-time requires shrinking dozens of denoising steps to a few. Standard DMD causes the generator to drift from the source video during long autoregressive rollouts. SA-DMD lets the teacher additionally attend to source-video frames strictly time-aligned with the current chunk during distillation, with classifier-free guidance splitting text-instruction strength and source-fidelity strength into two independent controls. Crucially, source anchoring happens only in the training-time teacher objective — deployed inference uses a single branch with no extra source computation. Like a copyist whose teacher checks the original painting during lessons, so the student's "muscle memory" resists drift. Result: denoising compressed to two steps.
3. LHAD (Long-Horizon Autoregressive Distillation)
Training on seconds-long clips never exposes the error states that appear after a minute or hour of streaming. LHAD splits a rolling sequence of m chunks into short contiguous segments, computes SA-DMD gradients per segment, frees the computation graph immediately (avoiding OOM), and accumulates gradients before a single update. A dynamic mirror looping trick (forward-then-reverse playback) extends source videos that are too short, avoiding abrupt looping artifacts.
Results
OpenVE-Bench (short video), Gemini as judge, max 5:
| Method | Params | Resolution | Score | |---|---|---|---| | StreamDiffusionV2 | 1.3B | 480×832 | 1.23 | | SANA-Streaming | 2B | 704×1280 | 2.62 | | LiveEdit | 1.3B | 480×832 | 2.00 | | XMax-X2.0 | — | 832×1440 | 1.87 | | Bernini-R (offline) | 27B | 480×848 | 3.72 | | JoyAI-Video-Edit | 16B | 720×1280 | 3.60 |
LongV2VBench (new benchmark, 229 tasks, 1-minute videos):
| Method | Resolution | Throughput | Score | |---|---|---|---| | StreamDiffusionV2 | 480×832 | 18.07 FPS | 1.21 | | SANA-Streaming | 704×1280 | 14.51 FPS | 1.64 | | LiveEdit | 480×832 | 15.45 FPS | 1.23 | | XMax-X2.0 | 832×1440 | 20.90 FPS | 1.71 | | JoyAI-Video-Edit | 720×1280 | 30.19 FPS | 3.30 |
Ablations
| SA-DMD | LHAD | Score | |---|---|---| | ✗ | ✗ | 2.81 | | ✓ | ✗ | 3.23 | | ✗ | ✓ | 3.06 | | ✓ | ✓ | 3.30 |
SA-DMD mainly curbs visual degradation (global style 3.61→4.24, local edits 3.43→4.00); LHAD targets late-video instability. Together they give the best results on local add/edit/delete.
Human preference (pairwise blind votes)
Deployment: how 720p@30FPS is achieved
Single NVIDIA B200 GPU, per 8-frame chunk: VAE encode 22 ms, DiT denoising (2 steps) 185 ms, VAE decode 19 ms → 226 ms request-to-response, 266 ms with KV-cache build and pseudo-encoding ≈ 30.1 FPS.
Engineering: FP8 quantization, compiled + auto-tuned VAE, pipelined host/GPU execution, warm-up, persistent compilation artifacts, and memory pooling. Versus competitors (81-frame input, batch 1), JoyAI delivers the lowest end-to-end latency (2.68 s) and highest FPS (30.19) at the highest resolution, with VAE throughput of 200 FPS — evidence of whole-pipeline co-optimization rather than a single algorithmic trick.
Training data: two routes to pair scarcity
1. Keyframe-guided edit propagation: edit one representative frame with an image-editing model, then feed the edited frame + original video to an image+video→video model to propagate the edit while preserving motion and unedited content. 2. Latent-shared image-to-video generation: generate source and edited videos from two I2V branches sharing early-denoising latents (consistent motion/composition), conditioned on different images later to inject the edit.
Both are filtered for quality, edit correctness, change magnitude, and temporal coherence, with MLLMs verifying source/target differences and refining instruction text.
Takeaways
1. Long-video drift is finally quantified: the dedicated LongV2VBench shows one-minute editing quality can halve without targeted optimization — prior "real-time" claims mostly held only for sub-10-second clips. 2. Move inference cost into training: SA-DMD's "study hard before the exam, rely on memory during it" philosophy applies to other quality-vs-latency problems (live speech translation, real-time face reconstruction). 3. System-level optimization matters: 30 FPS comes from co-optimizing VAE, DiT, caching, and quantization — every 5 ms shaved adds up. 4. Open problems: 30 FPS still requires 16B parameters and a B200. Real democratization of live editing awaits consumer-grade hardware.