LACE: Layer-Adaptive Codec Encoding for Dynamic Frame Rate Neural Audio Codecs

Forum topic · 小凯 · 2026-09-17

Summary

LACE (Layer-Adaptive Codec Encoding) is a dynamic frame rate neural audio codec that addresses the high computational cost of high-frame-rate codecs used in speech language modeling. Unlike prior dynamic frame rate approaches that apply a single compression step before multi-layer quantization or operate only on single-codebook codecs, LACE applies an independent compression step at each quantization layer, enabling layer-specific segmentation boundaries that reflect the differing rates of temporal change in residual embeddings. To make LACE tokens usable in downstream text-to-speech (TTS), the authors introduce a joint alignment and boundary anchoring mechanism that keeps durations consistent across layers while preserving compression benefits. Experiments on LibriTTS show LACE achieves better rate-quality trade-offs in reconstruction than previous dynamic frame rate methods and improves TTS inference efficiency while maintaining competitive synthesis quality. The code is released as part of the ESPnet3 codec recipes (arXiv:2609.17509).

Paper Overview

Field: NLP / Speech Authors: Thanapat Trachu, Samuele Cornell, William Chen, Shinji Watanabe Published: 2026-09-15 arXiv: 2609.17509

Abstract (Full Translation)

Neural audio codecs are a key component in speech language modeling. However, their high frame rates lead to long sequence lengths, increasing computational costs. Dynamic frame rate codecs mitigate this by reducing the effective frame rate using a compression step to merge multiple frames together. However, most prior methods either operate on single-codebook codecs or apply a single compression step before multi-layer quantization. This forces all quantization layers to share the same segmentation boundaries, despite the residual embeddings at different quantization layers exhibiting different rates of change over time.

We propose LACE (Layer-Adaptive Codec Encoding), a dynamic frame rate codec that applies an independent compression step at each quantization layer, enabling layer-specific segmentation boundaries. To use LACE tokens for downstream text-to-speech (TTS), we additionally introduce a joint alignment and boundary anchoring mechanism that keeps the durations of each layer consistent while retaining the benefits of compression. Experiments on LibriTTS show that LACE provides better rate-quality trade-offs on reconstruction than prior dynamic frame rate methods, and improves TTS inference efficiency while maintaining competitive synthesis quality. Our code has been released as part of the ESPnet3 codec recipes.

Key Takeaways

Problem: High frame rates in neural audio codecs inflate sequence lengths and computational costs for speech language modeling.
Limitation of prior work: Single compression step before multi-layer quantization forces shared segmentation boundaries across quantization layers, ignoring differing temporal dynamics of residual embeddings.
Contribution: LACE applies independent compression per quantization layer, plus a joint alignment and boundary anchoring mechanism for consistent durations in TTS.
Results: Better rate-quality trade-off on LibriTTS reconstruction and more efficient TTS inference with competitive quality.
Code: Available in the ESPnet3 codec recipes.