LACE: Layer-Wise Compression for Dynamic Frame Rate Neural Audio Codecs

Forum topic · 小凯 · 2026-09-17

Summary

LACE (Layer-Adaptive Codec Encoding) is a dynamic frame rate neural audio codec proposed by Thanapat Trachu, Samuele Cornell, William Chen, and Shinji Watanabe (arXiv:2609.17509). Neural audio codecs are essential for speech language modeling, but their high frame rates increase sequence length and computational cost. Existing dynamic frame rate codecs reduce effective frame rate by merging frames, yet most apply a single compression step shared across all quantization layers or only work with single-codebook codecs. LACE instead applies an independent compression step at each quantization layer, allowing layer-specific segmentation boundaries that match the different temporal rates of change of residual embeddings. A joint alignment and boundary anchoring mechanism keeps durations consistent across layers for downstream text-to-speech while retaining compression benefits. Experiments on LibriTTS show better rate-quality trade-offs than prior dynamic frame rate methods and improved TTS inference efficiency with competitive synthesis quality. The code is released as part of ESPnet3 codec recipes.

Paper Overview

Field: Speech / NLP Authors: Thanapat Trachu, Samuele Cornell, William Chen, Shinji Watanabe Published: 2026-09-15 arXiv: 2609.17509

Abstract (translated)

Neural audio codecs are a key component in speech language modeling. However, their high frame rates lead to long sequence lengths, increasing computational costs. Dynamic frame rate codecs mitigate this by reducing the effective frame rate using a compression step to merge multiple frames together. However, most prior methods either operate on single-codebook codecs or apply a single compression step before multi-layer quantization. This forces all quantization layers to share the same segmentation boundaries, despite the residual embeddings at different quantization layers exhibiting different rates of change over time.

The authors propose LACE (Layer-Adaptive Codec Encoding), a dynamic frame rate codec that applies an independent compression step at each quantization layer, enabling layer-specific segmentation boundaries. To make LACE tokens usable for downstream text-to-speech (TTS), they additionally introduce a joint alignment and boundary anchoring mechanism that keeps durations consistent across layers while preserving the compression benefits.

Experiments on LibriTTS show that LACE provides better rate-quality trade-offs on reconstruction than previous dynamic frame rate approaches, and improves TTS inference efficiency while maintaining competitive synthesis quality. The code has been released as part of the ESPnet3 codec recipes.

Key Contributions

Layer-wise compression: an independent compression step per quantization layer instead of one shared segmentation boundary.
TTS compatibility: joint alignment and boundary anchoring keeps per-layer durations aligned without sacrificing compression gains.
Results: better rate-quality trade-off on LibriTTS reconstruction and faster TTS inference with competitive quality.