YouDub-webui: A 9-Stage Open-Source Pipeline for AI Video Dubbing

Forum topic · QianXun · 2026-06-17

Summary

YouDub-webui is an open-source (Apache-2.0) video localization tool built by a Chinese creator with 800,000+ Bilibili followers, who has used it to dub more than 20,000 videos. The system chains nine stages into a single-process, serial FastAPI pipeline: yt-dlp download, Demucs vocal/BGM separation, Whisper large-v3-turbo transcription with word-level timestamps, LLM-based sentence segmentation and two-stage translation (whole-video preprocessing for context and glossary, then 50-thread concurrent per-sentence translation), zero-shot voice cloning and synthesis with VoxCPM2, pitch-preserving time-stretch alignment, and FFmpeg final muxing. Every stage caches its output to disk, enabling resume-from-failure without redoing upstream work. Task state lives in a local SQLite database with a Next.js frontend, requiring no message queue or cloud services. Compared with SaaS tools like HeyGen and Rask.ai, YouDub keeps all data local and costs only API tokens plus GPU time (an 8GB NVIDIA card suffices). Known limitations include single-task serial execution, one shared cloned voice for all speakers, burned-in subtitles, and CPU inference being impractical. The project demonstrates that full video dubbing can run offline on a personal computer at acceptable quality.

Key points

YouDub-webui is an open-source (Apache-2.0) video dubbing/localization tool: feed it a YouTube or Bilibili URL and it outputs a fully dubbed, subtitled video with no manual steps in between. The author (Bilibili creator "黑纹白斑马", 800k+ followers) has dubbed 20,000+ videos with it — it is a production system, not a demo.
Architecture: single process, single worker thread, serial FIFO pipeline. Next.js frontend (:3000) → FastAPI backend (:8000) → Python queue.Queue daemon thread → 9-stage pipeline. No message queues, no microservices — deliberately simple because there is only one GPU.

The 9-stage pipeline

| # | Stage | Tool | Input → Output | |---|-------|------|----------------| | 1 | Download | yt-dlp | URL → MP4 | | 2 | Separate | Demucs htdemucs_ft | audio → vocals.wav + BGM.wav | | 3 | Transcribe | Whisper large-v3-turbo | vocals → word-timestamped JSON | | 4 | Sentence split | ASR Sentence Fixer | Whisper segments → sentence boundaries | | 5 | Translate | OpenAI-compatible API | source text → target-language text | | 6 | Audio slicing | pydub + librosa | translation + reference audio → per-sentence WAVs | | 7 | Dub | VoxCPM2 | translated subtitles + reference audio → dubbed WAVs | | 8 | Mix | audiostretchy + pydub | dub + BGM → full dubbed track | | 9 | Mux | FFmpeg | video + dub + subtitles → final .mp4 |

Every stage's output is cached to disk. If translation (stage 5) fails, a rerun does not re-download the video or re-run Whisper — "cache as fault tolerance." Two execution modes: fully automatic (stop on error) and manual (pause after each stage for review).

Notable technical decisions

Separate first, then transcribe. Demucs htdemucs_ft splits audio into stems; vocals go to Whisper, BGM is preserved and remixed at the end. word_timestamps=True in Whisper provides the millisecond-level time skeleton that makes dub alignment possible.
Two-stage LLM translation. Stage 1 feeds the full transcript to the LLM to produce a summary, a glossary of terms (consistency across the whole video), and corrections of ASR homophone errors. Stage 2 translates sentence-by-sentence with ThreadPoolExecutor concurrency of 50 (I/O-bound), using a detailed prompt: keep colloquial force, follow the glossary, convert number formats ("1.5 million" → "150万"), use full-width punctuation, preserve hesitations and repetitions.
VoxCPM2 for voice cloning. OpenBMB's 2B-parameter, tokenizer-free TTS (trained on 2M+ hours, 30 languages + 9 Chinese dialects, Apache-2.0) generates speech directly as continuous representations decoded by an AudioVAE to 48 kHz, avoiding quantization loss and yielding more natural prosody.
Time alignment in two steps: a global speed factor per speaker (ratio of total dub duration to source duration), then per-sentence pitch-preserving time-stretching with audiostretchy (time stretch, not resampling — pitch must not change).

Engineering aesthetics

SQLite over PostgreSQL: three tables (tasks, task_stages, settings), zero external services — creators run just uvicorn + npm run dev.
Observability without Datadog: per-stage progress, status, duration, and errors persisted in SQLite and polled every 2 seconds by the frontend; logs streamed live via StreamingResponse.
vs. SaaS (HeyGen / Rask.ai): local-first means data never leaves the machine and costs are GPU hardware plus OpenAI API tokens (dubbing a 30-min video costs under ¥2 in translation tokens), at the price of setup friction (NVIDIA GPU with ≥8GB VRAM, CUDA, FFmpeg, yt-dlp).

Limitations and future work

GPU is a hard requirement (VoxCPM2 on CPU takes tens of seconds per sentence).
Serial FIFO execution processes one video at a time — fine for a solo creator, a bottleneck for teams.
One cloned voice for the whole video; multi-speaker voice separation is not yet supported.
Subtitles are hard-burned via FFmpeg (not switchable); lip-sync and per-sentence translation editing remain future work; a full Docker image would ease installation.

References

1. YouDub-webui — https://github.com/liuzhao1225/YouDub-webui 2. VoxCPM2 (OpenBMB) — https://github.com/OpenBMB/VoxCPM 3. Whisper — Radford et al., OpenAI, 2022. arXiv:2212.04356 4. Demucs — Défossez et al., Meta AI, 2023. arXiv:2211.08536 5. VoxCPM2 review — https://www.wangjun.dev/2026/06/openbmb-voxcpm2-tts/

*Based on a June 2026 source-code audit of YouDub-webui. Versions will iterate, but the logic behind the architectural decisions — simplicity, local-first, cache-as-fault-tolerance — outlasts version numbers.*