browser-use/video-use: LLMs Don't Watch Video, They Read Video

Key points

• The core thesis: LLMs should *read* video rather than *watch* it. Naive frame-by-frame ingestion of a 20-minute 1080p 30fps clip yields ~36,000 frames (~54M tokens, ~$810 on GPT-5); video-use reduces this to a 12KB packed transcript plus a few on-demand PNG composites (<$0.01).
• Two-layer abstraction:
• *Layer 1 (always loaded)*: ElevenLabs Scribe returns word-level timestamps, speaker labels, and audio events, packed into a takes_packed.md (~12KB) such as [002.52-005.36] S0 Ninety percent of what a web agent does is completely wasted.
• *Layer 2 (on-demand)*: a timeline_view PNG combining filmstrip, waveform, and word labels, generated only at decision points (ambiguous pauses, retake comparisons, cut sanity checks). A 20-minute video may need only 5–10 such images.
• Shared philosophy with browser-use: browser-use gives the LLM a structured DOM instead of screenshots; video-use gives it a word-level transcript instead of frames. Both compress high-dimensional perceptual data into low-dimensional symbolic representations so the LLM can lean on its text-reasoning strengths.
• Pipeline: Transcribe → Pack → LLM Reasons → EDL → Render → Self-Eval with up to three automatic re-renders if issues are detected.
• Self-Eval at every cut boundary: visually checks for visual jumps, audio pops, and hidden captions — treating verification as a built-in component, not a post-hoc filter.
• Skill rules as alignment: SKILL.md defines 12 hard rules, including "Text + on-demand visuals," "Audio is primary, visuals follow," "Ask → confirm → execute → self-eval → persist," and "Production-correctness is non-negotiable. Taste isn't." Markdown rules constrain behavior without retraining.
• EDL pattern: the LLM outputs a structured Edit Decision List; deterministic code (ffmpeg) executes it — analogous to a "LLM as poet, deterministic engine as accountant" split.
• Why it works: LLM text reasoning dominates visual reasoning (e.g., GPT-5 at ~92% on MMLU vs. weaker performance on many vision benchmarks), so routing editing decisions through transcripts hits the model's strengths.
• Practical specs: depends on ffmpeg (required), yt-dlp (optional), and the ElevenLabs API (transcription); works with Claude Code, Codex, Hermes, Openclaw, or any agent with shell access; 100% open source.
• Takeaways for agent builders:

1. Dimensionality reduction is the core strategy for letting LLMs handle high-dimensional media — swap modality surfaces, don't try to make the model "see better." 2. Self-Evaluation at each output boundary is required for production-grade agents. 3. Structured decision lists (EDLs) are safer than letting the LLM emit raw tool commands. 4. A skill file of hard rules can deliver alignment without retraining.

> Repository: https://github.com/browser-use/video-use > Try it in the browser: https://browser-use.com