Context-Aware Interleaved Batching for WhisperX: Faster, More Accurate Speech Transcription (arXiv 2509.00138)

Forum topic · 小凯 · 2026-09-02

Summary

A new paper (arXiv:2509.00138) by Carlos Bain and Max Bain, published September 1, 2025, proposes Context-Aware Interleaved Batching to improve WhisperX-based speech transcription. WhisperX accelerates transcription via intra-audio batching, but isolating audio segments loses the historical context needed for coherent punctuation and terminology transcription. Standard Whisper preserves context sequentially but suffers from slow inference and hallucination loops. The proposed method uses VAD-derived segment boundaries to stabilize Whisper's text conditioning, enabling continuous historical context to be maintained safely across batched audio segments. Experiments on long-form audio benchmarks show reduced Word Error Rate (WER), improved proper noun transcription, and preserved high-throughput inference speeds, combining the accuracy of sequential decoding with the throughput of batching.

Paper Overview

Field: NLP / Speech Transcription
Authors: Carlos Bain, Max Bain
Published: 2025-09-01
arXiv: 2509.00138

Background

WhisperX accelerates speech transcription through intra-audio batching, but this isolates audio segments and discards the historical context needed for coherent punctuation and consistent terminology transcription. In contrast, standard Whisper processes audio sequentially and retains context, but suffers from slow inference and hallucination loops.

Proposed Method

The paper introduces Context-Aware Interleaved Batching, which aims to combine the strengths of both approaches:

Uses VAD-derived segment boundaries to stabilize Whisper's text conditioning
Safely maintains continuous historical context across batched audio segments

Results

Experiments on long-form audio benchmarks demonstrate that the approach:

Reduces Word Error Rate (WER)
Improves proper noun transcription
Maintains high-throughput inference speeds

Original Abstract

> While WhisperX accelerates speech transcription via intra-audio batching, it isolates audio segments, losing the historical context needed for coherent punctuation and terminology transcription. Conversely, standard Whisper retains context sequentially but suffers from slow inference and hallucination loops. To achieve the best of both worlds, we propose Context-Aware Interleaved Batching. By using VAD-derived segment boundaries, our algorithm stabilizes Whisper's text conditioning, allowing us to safely maintain continuous historical context across batched audio segments. As demonstrated on long-form audio benchmarks, this approach reduces Word Error Rate (WER) and improves proper noun transcription, all while maintaining high-throughput inference speeds.

---

Paper link: arXiv:2509.00138