Context-Aware Interleaved Batching for WhisperX: Faster, More Accurate Long-Form Speech Transcription

Forum topic · 小凯 · 2026-09-02

Summary

This post summarizes arXiv paper 2509.00138 (September 1, 2025) by Carlos Bain and Max Bain, which introduces Context-Aware Interleaved Batching for WhisperX. WhisperX accelerates speech transcription through intra-audio batching but isolates audio segments, losing historical context needed for coherent punctuation and consistent terminology. Standard Whisper retains context sequentially but suffers from slow inference and hallucination loops. The proposed method uses VAD-derived segment boundaries to stabilize Whisper's text conditioning, enabling safe maintenance of continuous historical context across batched audio segments. Experiments on long-form audio benchmarks show the approach reduces Word Error Rate (WER), improves proper noun transcription, and preserves WhisperX's high-throughput inference speeds.

Paper Overview

Field: NLP (speech transcription)
Authors: Carlos Bain, Max Bain
Published: 2025-09-01
arXiv: 2509.00138

Abstract

While WhisperX accelerates speech transcription via intra-audio batching, it isolates audio segments, losing the historical context needed for coherent punctuation and terminology transcription. Conversely, standard Whisper retains context sequentially but suffers from slow inference and hallucination loops. To achieve the best of both worlds, we propose Context-Aware Interleaved Batching. By using VAD-derived segment boundaries, our algorithm stabilizes Whisper's text conditioning, allowing us to safely maintain continuous historical context across batched audio segments. As demonstrated on long-form audio benchmarks, this approach reduces Word Error Rate (WER) and improves proper noun transcription, all while maintaining high-throughput inference speeds.

Key Takeaways

The problem: WhisperX's batching isolates segments, degrading punctuation and terminology consistency; sequential Whisper is slow and prone to hallucination loops.
The solution: Context-Aware Interleaved Batching keeps continuous historical context across batched segments, stabilized by VAD-derived segment boundaries.
Results: Lower WER, better proper noun transcription, and unchanged high-throughput inference on long-form audio benchmarks.