Context-Aware Interleaved Batching for WhisperX: Faster Whisper Transcription Without Losing Context

Forum topic · 小凯 · 2026-09-02

Summary

A new paper (arXiv:2509.00138, 2025-09-01) by Carlos Bain and Max Bain proposes Context-Aware Interleaved Batching to improve WhisperX speech transcription. WhisperX accelerates transcription through intra-audio batching, but batching isolates audio segments and discards the historical context needed for coherent punctuation and consistent terminology. Standard Whisper preserves context sequentially but is slow and prone to hallucination loops. The proposed method uses VAD-derived segment boundaries to stabilize Whisper's text conditioning, enabling continuous historical context to be safely maintained across batched audio segments. Experiments on long-form audio benchmarks show the approach reduces Word Error Rate (WER), improves proper noun transcription, and maintains high-throughput inference speeds.

Overview

Field: NLP Authors: Carlos Bain, Max Bain Published: 2025-09-01 arXiv: 2509.00138

Abstract

While WhisperX accelerates speech transcription via intra-audio batching, it isolates audio segments, losing the historical context needed for coherent punctuation and terminology transcription. Conversely, standard Whisper retains context sequentially but suffers from slow inference and hallucination loops. To achieve the best of both worlds, we propose Context-Aware Interleaved Batching. By using VAD-derived segment boundaries, our algorithm stabilizes Whisper's text conditioning, allowing us to safely maintain continuous historical context across batched audio segments. As demonstrated on long-form audio benchmarks, this approach reduces Word Error Rate (WER) and improves proper noun transcription, all while maintaining high-throughput inference speeds.

Key Takeaways

Problem: WhisperX's intra-audio batching is fast but processes segments in isolation, degrading punctuation and terminology consistency.
Trade-off: Sequential Whisper keeps context but is slower and prone to hallucination loops.
Solution: Context-Aware Interleaved Batching leverages VAD-derived segment boundaries to stabilize text conditioning and carry continuous historical context across batched segments.
Results: Lower WER, better proper noun transcription, and high-throughput inference on long-form audio benchmarks.