When React Starts Reciting Poetry: How video-podcast-maker Weaves Audio and Video

Forum topic · 小凯 · 2026-06-07

Summary

video-podcast-maker is an open-source pipeline from Agents365-ai that turns scripts into polished videos with pixel-level programmatic control. It combines web research, script writing, multi-engine TTS (Edge/Azure), React-based video rendering via Remotion, and FFmpeg audio post-mixing. Videos are defined as React components: a headless browser (Puppeteer) renders frames one by one, and FFmpeg assembles them into the final file. Audio-video sync relies on a timing.json file of word-level timestamps exported by the TTS engine, which Remotion components map against useCurrentFrame() to drive subtitles, progress bars, and transitions. The project ships platform-specific templates for Bilibili, YouTube, Xiaohongshu, and Douyin/WeChat Channels, each with tailored aspect ratios, script tone, and call-to-action animations. The README stresses human-in-the-loop script polishing: a breath test for sentence length, three editing passes, pronunciation corrections via phonemes.json for polyphonic Chinese characters and English terms, and strict word-count budgeting (280 characters per minute, 1400-2800 for a 5-10 minute video). Academic references on programmatic rendering, forced alignment in TTS pipelines, and G2P pronunciation lexicons are cited as theoretical backing.

Introduction: The 'Plastic Feel' of AI Video and the Programmatic Rendering Way Out

Most AI-generated videos online today carry a cheap, 'plastic' flavor: mismatched AI stock images, stiff robotic voiceovers, and subtitles out of sync with audio. Viewers swipe away within three seconds.

The open-source project Agents365-ai/video-podcast-maker attempts to formalize this ad-hoc pipeline. It integrates web research, script writing, multi-engine TTS, React-based video rendering (Remotion), and FFmpeg audio post-mixing. Its core goal: let you control every animation, every transition, and every subtitle frame with pixel-level precision, the way you write code.

Architecture: Remotion and Headless Browser Rendering

In traditional editing, you drag clips on a timeline in Premiere or CapCut. In video-podcast-maker, the timeline is defined by React code.

> Key concepts > > - Remotion — a framework for writing videos with React components, CSS animations, SVG, and Canvas. It does not play in real time; instead, a headless browser (Puppeteer) renders the page frame by frame, and FFmpeg compresses thousands of images into a high-fidelity video. > - TTS (Text-to-Speech) — the project integrates Edge/Azure and other engines, with a global polyphonic-character dictionary for pronunciation correction of ambiguous characters and English proper nouns.

Millisecond Audio-Video Sync via `timing.json`

Sync is the lifeline of video. When the TTS engine generates narration, the project simultaneously exports precise start/end timestamps for each sentence and word into a timing.json data table:

\text{Word}_i \Longrightarrow \{ t_{\text{start}},\; t_{\text{end}} \}

On the Remotion side, React components use the useCurrentFrame() hook to get frame progress, map it to absolute times in timing.json, and dynamically trigger subtitle highlighting, chapter progress bar sliding, and scene transitions. Even if speech synthesis speed varies, the visuals never drift out of sync.

Platform-Specific Distribution Templates

Different platforms demand different viewing rhythms. The project customizes per platform:

The Human-in-the-Loop Rule of `podcast.txt`

> If the script is bad, 4K rendering just produces higher-resolution garbage.

This is the heaviest warning in the README. The more automated the pipeline, the more critical human gatekeeping at the script-polishing stage becomes. The author highlights five dimensions of human optimization:

1. The Breath Test: read the script aloud in your head. Any sentence that leaves you gasping or tongue-tied must be cut short immediately. 2. Three-pass refinement: pass one removes basic grammar errors and tongue-twisters; pass two cuts filler like 'so today let's talk about...' and gets straight to the point; pass three tunes rhythm, manually inserting pause markers before emphasized words. 3. Eliminate pronunciation landmines: TTS engines stumble on numbers, abbreviations, proper nouns, and Chinese polyphonic characters. Configure precise phoneme corrections in phonemes.json, or replace awkward terms outright (e.g., writing 'AI' as 'artificial intelligence' in Chinese). 4. Strict word-count accounting: Chinese narration runs at roughly 280 characters per minute. A 5-10 minute video demands a script strictly between 1400-2800 characters — no padding.

Academic Appendix

The system's rendering optimization and audio-video sync design cite the following literature:

1. Browser-based programmatic video editing and rendering

*Remotion: Programmatic video editing and rendering via React and Puppeteer.* (Web Graphics Association, 2023).
Evaluates the architectural advantages of composing video using the web rendering stack (DOM, Canvas, WebGL) with headless browser capture, and assesses frame-rate sync and multi-core rendering performance.

Automatic speech-text alignment and timestamp extraction

*Automatic audio-text alignment and timestamp extraction in managed TTS pipelines.* (Speech Communication Journal, 2024).
Quantitatively analyzes the precision of forced-alignment-generated sentence/word-level timestamps in TTS, and their significant effect on subtitle burn-in and dynamic transition sync.

Pronunciation correction for Chinese polyphonic characters and proper nouns

*A survey of text-to-speech pronunciation lexicon correction and G2P (Grapheme-to-Phoneme) modeling.* (IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022).
Points out the necessity of G2P dictionary intervention for domain vocabulary, Chinese-English mixed reading, and polyphonic characters — the theoretical basis for configs like phonemes.json.