AI Video Studio Architecture Deep Dive: A Local-First, Scene-Based Video Workbench from Script to Final Cut

Overview

AI Video Studio Community Edition is a local-first, scene-based video workbench for knowledge creators who need to turn written scripts into finished videos. Unlike one-click AI video tools (Runway, Pika, Kling), it gives directors fine-grained control over pacing, narration alignment, subtitles, and motion graphics.

• Project: AI Video Studio Community Edition
• Author: WoyouWoyou
• License: AGPL-3.0 (commercial use requires separate licensing)
• Code: https://github.com/WoyouWoyou/ai-video-studio
• Stack: FastAPI + React + Vite + GSAP + HyperFrames

Key points

• Scene-based abstraction: Each scene contains a background (AI image or HTML template), TTS narration with word-level timestamps, word-highlight subtitles, MG overlays (title cards, bullet lists, progress bars), and per-element timing.
• Single data model: Everything operates on one Pydantic v2 Plan JSON schema — no database, no ORM; projects are a plan.json plus an assets folder, versionable with Git.
• Automatic timeline resolution: resolve_timeline() derives scene durations from audio length, allocates segment times weighted by text length, and aligns overlay reveal times to their segments minus a reveal_lead — editing narration auto-recomputes the whole timeline.
• Provider registry pattern: All AI services plug in via registries (ImageRegistry, TtsRegistry, HtmlRegistry); implement a BaseProvider interface and register it. Currently includes Gemini, Bailian Wanx, and Stable Diffusion for images, Edge TTS for voice.
• Human-in-the-loop script review: AI-split scripts produce a ScriptReviewDraft that users must approve, edit, or discard before scenes are generated.

Architecture

• Frontend (React + Vite + GSAP): three-pane layout — scene list, RAF-driven preview player with adaptive scaling via ResizeObserver, and a property inspector. State via three Zustand stores (useProjectStore with 800ms debounced autosave, usePlayerStore, useUiStore).
• Backend (FastAPI + Pydantic v2): PlanStore for JSON file CRUD, service layer (SceneService, ScriptService, MediaService), and provider registries. Export runs as an async job (queued → running → success/failed) with progress polling.
• Local-first storage: projects live in studio-projects/<project_id>/ with plan.json, assets/, and exports/.

Web-to-Video export pipeline

The V11Bridge serializes the Plan into a renderable web project: each scene becomes an isolated HTML file, and an index.html composes them as iframes driven by a GSAP timeline, syncing local time to each iframe via postMessage ({ type: "studio:set-time", time }). Benefits: WYSIWYG (preview and render use the same HTML/CSS/GSAP), renderer-agnostic, debuggable in a browser. Streaming encode is disabled in favor of a capture-then-encode path (frame-by-frame screenshots, then FFmpeg encoding), which is slower but more stable for iframe composition. Note: the HyperFrames CLI itself is not open source — a key dependency risk.

Limitations

1. HyperFrames CLI dependency is not open source. 2. Animation templates are limited to simple effects (fade, rise, zoom); complex MG animation requires hand-written HTML/CSS. 3. No AI video generation (Runway/Kling) as backgrounds yet — images only. 4. Desktop build requires users to install FFmpeg and HyperFrames themselves.

Design philosophy

Local-first, scene-based thinking, human-in-the-loop, pluggable providers, and cost control (free Edge TTS, low-cost image generation, no subscriptions). Roughly 10k lines of code, with clear architecture — worth studying if you're building AI video tooling.

References

• GitHub: https://github.com/WoyouWoyou/ai-video-studio
• Quark netdisk: https://pan.quark.cn/s/3e62fff831e3 (code: iycv)
• China Mobile cloud: https://yun.139.com/shareweb/#/w/i/2v3EhsXDCvfn2 (code: k75g)