JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion (Paper Review)

Forum topic · 小凯 · 2026-08-22

Summary

JoyAI-Video-Edit, from JD's Joy Future Academy (arXiv 2608.03974), is a 16-billion-parameter model that performs open-ended, instruction-driven video editing in real time at 720p@30FPS, enabling live-stream editing without knowing the video length in advance. The paper tackles the core conflict between offline editors (high quality via bidirectional attention, but requiring the full clip) and streaming editors (real-time but with severe quality loss and long-term drift). Its three key techniques are: (1) chunk-wise causal autoregressive diffusion with a sliding window plus global anchor and Resampling Forcing to mitigate train-inference mismatch; (2) SA-DMD, a source-anchored distribution-matching distillation that compresses denoising to two steps while anchoring edits to the source video during teacher training only; and (3) LHAD, long-horizon autoregressive distillation for stability over minute-long videos. On OpenVE-Bench it scores 3.60, nearly matching the offline 27B Bernini-R (3.72), and on its new LongV2VBench it scores 3.30, nearly double the best streaming baseline, at 30.19 FPS on a single B200 GPU with FP8 and pipeline optimizations.

JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion — Paper Review

> Paper: JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion > arXiv: 2608.03974 | Team: JD Joy Future Academy | Category: cs.CV > One-liner: 16B parameters, 720p@30FPS, real-time editing of arbitrarily long video streams — turning video editing from post-production into a live, in-stream process.

Why real-time video editing is hard

Imagine live-streaming yourself kicking a ball in a desert, viewers typing "make it a rainy stadium," and the scene transforming within a second — rain, puddles, wet clothing — all rendered live. This is the target scenario: editing that happens during the stream, not after recording.

Before this paper, video editing lived in two separate worlds:

Offline editors (VACE, OpenVE-Edit, Bernini-R): bidirectional attention over the full clip with iterative denoising. Excellent quality (3.6+ / 5 on OpenVE-Bench), but unusable for live streams.
Streaming/real-time editors (StreamDiffusionV2, SANA-Streaming, LiveEdit, XMax-X2.0): causal generation frame by frame, but quality scores only 1.2–2.6 — blur, color drift, poor instruction following.

The core question: can one model achieve offline-level quality and live-stream responsiveness?

The train–inference mismatch problem

Naively causalizing an offline model fails because of teacher forcing: during training the model sees clean ground-truth history, but at inference it consumes its own imperfect generations. Errors compound like a game of telephone — long-term drift in color, identity, and background. Video editing is harder still, since unedited regions must stay faithful to the source video while instructions are executed consistently. Too much reliance on generated history causes drift; too much on the source causes incomplete edits. A dynamic balance is needed.

The three core techniques

1. Chunk-wise autoregressive diffusion

Architecture: an MLLM for instruction + first-frame conditioning, a causal VAE (8×24×24 compression, 8 frames → 1 latent frame), and an MM-DiT backbone.
Videos are processed in chunks with bidirectional attention inside each chunk and causal attention across chunks.
A sliding window + global anchor design keeps the first chunk plus recent chunks as context, so memory cost is fixed regardless of stream length — like a simultaneous interpreter who only needs the opening tone plus the last few minutes.
Resampling Forcing: each training history chunk is first passed through a single denoising step to create imperfect, model-generated history (no gradients), so the model trains on realistic deployment conditions.

2. SA-DMD (Source-Anchored Distribution Matching Distillation)

Real-time requires compressing dozens of denoising steps to a few via DMD-style distillation, but naive DMD causes hallucination and source deviation over long autoregressive runs.
SA-DMD lets the teacher additionally attend to the temporally aligned source video during distillation, with classifier-free guidance splitting text-instruction and source-fidelity strength into independent controls.
Crucially, source anchoring is applied only in the teacher's training target — at deployment the model runs a single branch with no extra source computation. Analogy: a copyist trained while the teacher glances at the original painting internalizes a habit of not drifting.
Result: denoising compressed to two steps.

3. LHAD (Long-Horizon Autoregressive Distillation)

Training only on short clips leaves models unprepared for deep error states that emerge after 50+ seconds.
LHAD splits a long rolling sequence into short contiguous segments, computes SA-DMD gradients per segment, clears the computation graph (avoiding OOM), and accumulates gradients for one update.
Dynamic mirror looping (forward–backward alternation) extends source videos beyond their original length without jarring repeats.

Results

Short-video editing (OpenVE-Bench, Gemini-judged, 5-point scale)

| Method | Params | Resolution | Score | |---|---|---|---| | StreamDiffusionV2 | 1.3B | 480×832 | 1.23 | | SANA-Streaming | 2B | 704×1280 | 2.62 | | LiveEdit | 1.3B | 480×832 | 2.00 | | XMax-X2.0 | — | 832×1440 | 1.87 | | Bernini-R (offline) | 27B | 480×848 | 3.72 | | JoyAI-Video-Edit | 16B | 720×1280 | 3.60 |

Best streaming model by ~1 point; first place in four of five task categories; ties best on local edits (4.47) and nearly matches the offline 27B Bernini-R.

Long-video editing (LongV2VBench, 229 tasks, 1-minute videos, self-built)

| Method | Resolution | Throughput | Score | |---|---|---|---| | StreamDiffusionV2 | 480×832 | 18.07 FPS | 1.21 | | SANA-Streaming | 704×1280 | 14.51 FPS | 1.64 | | LiveEdit | 480×832 | 15.45 FPS | 1.23 | | XMax-X2.0 | 832×1440 | 20.90 FPS | 1.71 | | JoyAI-Video-Edit | 720×1280 | 30.19 FPS | 3.30 |

Score nearly 2× the runner-up while being fastest — baselines degrade sharply on long videos, confirming the drift problem.

Ablations

| SA-DMD | LHAD | Score | |---|---|---| | ✗ | ✗ | 2.81 | | ✓ | ✗ | 3.23 | | ✗ | ✓ | 3.06 | | ✓ | ✓ | 3.30 |

SA-DMD mainly fixes global style (3.61→4.24) and local edits; LHAD mainly stabilizes long-video tasks; together they are best on local add/edit/delete.

Human preference (head-to-head votes)

vs LiveEdit / SANA-Streaming / StreamDiffusionV2: 90% / 87% / 87%
vs XMax-X2.0: 81%
vs offline Bernini-R: 48% vs 44% — effectively a tie
vs closed-source Kling-3.0 Omni / Seedance 2.0: 56%

Deployment performance (single NVIDIA B200)

Per 8-frame chunk: VAE encode 22 ms, 2-step DiT denoise 185 ms, VAE decode 19 ms → 226 ms request-to-response, 266 ms with cache construction → ~30.1 FPS.

Optimizations: FP8 quantization, compiled VAE with auto-tuning, pipelined host/GPU execution, warm-up, persisted compilation artifacts, and memory pools. Versus competitors on 81-frame input, JoyAI achieves the lowest latency (2.68 s) and highest throughput (30.19 FPS) at the highest streaming resolution, with VAE throughput of 200 FPS — gains come from whole-system co-optimization, not a single component.

Training data: solving paired-data scarcity

Two pipelines generate (source, edited) video pairs:

1. Keypoint-guided edit propagation: edit one representative frame with an image editor, then propagate the edit across the video via an image+video→video model. 2. Latent-shared I2V generation: two I2V branches generate source and edited videos from an image pair, sharing latent variables early in denoising for consistent motion while diverging later via different conditioning.

Both pass quality filtering and MLLM-based verification to refine edit instructions.

Key takeaways

1. Long-video drift is finally quantified: without dedicated handling, 1-minute editing quality can halve versus 10-second clips — prior "real-time" results mostly held only for short clips. 2. Move inference cost into training: SA-DMD's "study with the teacher, perform from memory" philosophy generalizes to other real-time quality-critical tasks. 3. Systems matter: the 30 FPS result comes from co-optimized VAE, DiT, caching, and quantization; small per-stage savings add up. 4. Open problem: 30 FPS currently needs 16B parameters and a B200; consumer hardware remains a future milestone.

References

Yicheng Xiao, Wenxun Dai, et al. "JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion." arXiv:2608.03974, 2026.
JD Joy Future Academy: https://research.joyai.com/