JoyAI-Video-Edit: Real-Time Streaming Video Editing with Autoregressive Diffusion
Paper: JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion arXiv: 2608.03974 Team: JD Joy Future Academy Category: cs.CV
---
Why Real-Time Video Editing Is Hard
Before this work, video editing was split into two camps that could not talk to each other:
Offline editing (VACE, OpenVE-Edit, Bernini-R): The model reads the entire video using bidirectional attention and performs many denoising steps before outputting the result. Quality is excellent because global context is available, but the entire clip must be finished first — impossible for live streams.
Streaming / real-time editing (StreamDiffusionV2, SANA-Streaming, LiveEdit, XMax-X2.0): Frames arrive one at a time and the model only sees past frames (causal generation). Latency is low, but quality drops sharply. On OpenVE-Bench these methods score between 1.21 and 2.62 out of 5, while strong offline systems exceed 3.6.
The core question: can a model deliver offline-quality edits with live-stream latency?
---
Why Naive Causation Fails
The first impulse is to simply make an offline model causal — let it see only past frames. This triggers a train-inference mismatch: during training, the model is conditioned on ground-truth history (teacher forcing), but at inference it conditions on its own imperfect outputs. Errors accumulate frame after frame, producing long-term drift in color, identity, and background detail.
The task is doubly hard because video editing must stay aligned with the source video (untouched regions like actor motion must be preserved) while faithfully executing the edit prompt (e.g. "dress the protagonist as Iron Man"). Over-reliance on generated history causes drift; over-reliance on the source video weakens instruction following. A dynamic balance is required.
---
The Three-Pillar Solution
1. Chunk-wise Autoregressive Generation
The architecture stacks:
- MLLM: converts the natural-language instruction and the first frame into conditioning signals.
- Causal VAE: compresses 8 consecutive frames into one latent frame (8×24×24 compression).
- MM-DiT: the multimodal diffusion Transformer that performs the editing.
- vs LiveEdit: 90% preference
- vs SANA-Streaming: 87%
- vs StreamDiffusionV2: 87%
- vs XMax-X2.0: 81%
- vs Bernini-R (offline): 48% vs 44% — statistical tie with a top offline system
- vs Kling-3.0 Omni / Seedance 2.0 (commercial closed-source): 56%
- Yicheng Xiao, Wenxun Dai, et al. *JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion.* arXiv:2608.03974, 2026.
- JD Joy Future Academy: https://research.joyai.com/
The key change is chunk-wise processing: the video is sliced along the time axis. Within a chunk, the model uses bidirectional attention; across chunks it uses causal attention, so each chunk only sees previous chunks. A sliding window plus a fixed global anchor chunk (the first chunk) keeps the historical context bounded regardless of stream length.
Resampling forcing trains the model with self-generated, imperfect history rather than clean ground-truth history, so it becomes robust to its own past errors at inference time.
2. SA-DMD: Source-Anchored Distribution Matching Distillation
Standard Distribution Matching Distillation (DMD) compresses many denoising steps into a few, but in autoregressive video editing it can amplify drift. SA-DMD anchors the teacher's targets to the temporally aligned source-video frame during training only. Classifier-free guidance is split into two independent knobs: one for text instruction strength and one for source-video fidelity. The deployed model runs a single inference path — no extra source-video reference is needed at runtime. After SA-DMD the diffusion loop is reduced to two denoising steps.
3. LHAD: Long-Horizon Autoregressive Distillation
Models trained only on short clips have never encountered the deep accumulated errors of a multi-hour stream. LHAD splits a long rolling sequence of *m* chunks into shorter segments, computes SA-DMD gradients per segment, immediately frees each segment's computation graph to avoid OOM, and aggregates gradients for one parameter update. This exposes the model to long-horizon failure modes without exploding memory.
A small extra trick — dynamic mirror looping — alternates forward and reverse playback of source video to extend its effective length and avoid jarring loop artefacts.
---
Experimental Results
Short-video editing (OpenVE-Bench)
Five task categories (global style, local modification, background swap, local removal, local addition). Gemini as judge; maximum score is 5.
| Method | Parameters | Resolution | Overall | |--------|------------|------------|---------| | StreamDiffusionV2 | 1.3B | 480×832 | 1.23 | | SANA-Streaming | 2B | 704×1280 | 2.62 | | LiveEdit | 1.3B | 480×832 | 2.00 | | XMax-X2.0 | — | 832×1440 | 1.87 | | Bernini-R (offline) | 27B | 480×848 | 3.72 | | JoyAI-Video-Edit | 16B | 720×1280 | 3.60 |
JoyAI leads streaming methods on four of five categories, sits just 0.12 behind a 27B offline model, and ties for the top local-modification score (4.47).
Long-video editing (LongV2VBench — newly introduced)
229 tasks with one-minute videos — a regime previous benchmarks did not cover.
| Method | Resolution | Throughput | Overall | |--------|------------|------------|---------| | StreamDiffusionV2 | 480×832 | 18.07 FPS | 1.21 | | SANA-Streaming | 704×1280 | 14.51 FPS | 1.64 | | LiveEdit | 480×832 | 15.45 FPS | 1.23 | | XMax-X2.0 | 832×1440 | 20.90 FPS | 1.71 | | JoyAI-Video-Edit | 720×1280 | 30.19 FPS | 3.30 |
JoyAI's overall score is nearly double that of the runner-up; throughput is 44.4% higher than XMax-X2.0 despite a lower resolution.
Ablation
| SA-DMD | LHAD | Overall | |:------:|:----:|---------| | no | no | 2.81 | | yes | no | 3.23 | | no | yes | 3.06 | | yes | yes | 3.30 |
SA-DMD alone strongly suppresses visual artefacts (global style 3.61→4.24, local modification 3.43→4.00). LHAD alone targets late-stage instability in long videos. Together they cover complementary failure modes.
Human preference study
Two-way blind comparisons reported by the authors:
---
Deployment Performance on a Single NVIDIA B200
Per 8-frame chunk:
| Component | Time | |-----------|------| | VAE encode | 22 ms | | DiT denoise (2 steps) | 185 ms | | VAE decode | 19 ms | | Request-to-response | 226 ms | | With KV cache build + pseudo-encode | 266 ms | | Equivalent frame rate | ≈ 30.1 FPS |
Engineering levers listed by the authors: FP8 quantization, compiled and auto-tuned VAE, pipelined host–GPU overlap, persistent compile artefacts, and warm-start memory pools.
Head-to-head latency on 81-frame inputs at batch size 1:
| Method | Resolution | Latency | FPS | VAE FPS | |--------|------------|---------|-----|---------| | StreamDiffusionV2 | 480×832 | 4.48 s | 18.07 | 37.06 | | LiveEdit | 480×832 | 5.24 s | 15.45 | 37.26 | | SANA-Streaming | 704×1280 | 5.58 s | 14.51 | 27.12 | | JoyAI | 720×1280 | 2.68 s | 30.19 | 200.00 |
JoyAI simultaneously achieves the highest resolution in the comparison, the lowest latency, and a VAE throughput roughly five times the next-best competitor, indicating the advantage comes from full-pipeline optimization rather than a single trick.
---
Training Data: Solving the Pair-Scarcity Problem
High-quality (source-video, edited-target) pairs are rare. The authors pursue two pipelines:
1. Key-frame-guided edit propagation: pick one representative frame, edit it with an image-editing model, then feed the edited frame plus the original video into an image+video-to-video model that propagates the change while preserving motion and untouched regions.
2. Latent-shared image-to-video generation: from an original image and its edited counterpart, run two I2V branches that share latents during early denoising (for motion and layout consistency) and use different image conditions during later denoising (to introduce the edit).
Quality control filters samples on visual quality, edit correctness, content change extent, and temporal coherence, then uses a multimodal LLM to cross-check differences between source and edited videos and to refine the textual prompts. This auditor-style loop keeps the dataset aligned with natural-language supervision.
---
Takeaways
1. Long-horizon drift is a first-class problem. Most "real-time" editors score well on sub-10-second clips but collapse on one-minute streams. LongV2VBench is proposed precisely to expose this. 2. Pre-pay inference cost during training. SA-DMD bakes source-video grounding into the student model so the deployment path stays single-branch — a useful principle for any real-time generative system (speech translation, face reconstruction, etc.). 3. System co-design matters. VAE, DiT, cache, quantization, and compile-time choices combine; shaving 5 ms in each stage is the difference between 24 and 30 FPS. 4. Open frontier. Reaching 30 FPS still requires a 16B model and a B200 GPU. Until the same quality lands on consumer GPUs or phones, true mass-market live video editing remains a goal.
---