Geometric Iterative Retrieval for Neural Audio Codec Resynthesis

Forum topic · 小凯 · 2026-08-21

Summary

A new arXiv paper (2608.19141) by Leo Schmidt-Traub, Frédéric Berdoz, Luca A. Lanzendörfer, and Roger Wattenhofer introduces geometric iterative retrieval, a novel paradigm for resynthesizing high-quality audio from coarse codec tokens. Neural audio codecs based on Residual Vector Quantization (RVQ) are the dominant discrete representation for token-based audio generation, but resynthesis from these tokens limits overall system fidelity. The authors argue that the standard dichotomy between discrete token prediction and continuous regression is incomplete. Instead, their method treats the RVQ layer hierarchy as a natural iterative decomposition in continuous codebook space, performing contrastive retrieval in the codebook's geometric space rather than classifying over discrete vocabularies or regressing to a single target vector. Evaluated on codec restoration tasks for speech and music, the approach outperforms single-shot token prediction and one-step regression baselines, offering a path to higher-fidelity token-based audio generation systems.

Paper Overview

Field: Machine Learning Authors: Leo Schmidt-Traub, Frédéric Berdoz, Luca A. Lanzendörfer, Roger Wattenhofer Published: 2026-08-19 arXiv: 2608.19141

Abstract

Neural audio codecs based on Residual Vector Quantization (RVQ) have become the dominant discrete representation for token-based general audio generation, yet resynthesizing high-quality audio from coarse codec tokens remains an open problem and bounds the fidelity of every system that generates them.

Prior work has framed resynthesis as a choice between discrete token prediction and continuous regression. The authors argue that this dichotomy is incomplete and introduce geometric iterative retrieval, a paradigm that uses the RVQ layer hierarchy itself as a natural iterative decomposition in continuous codebook space. Rather than classifying over discrete vocabularies or regressing to a single target vector, the method performs contrastive retrieval in the codebook's geometric space.

Key Findings

RVQ-based neural audio codecs dominate token-based general audio generation, but coarse-token resynthesis limits system fidelity.
The proposed paradigm replaces the binary framing of token prediction vs. continuous regression with iterative retrieval over the RVQ codebook geometry.
The RVQ layer hierarchy provides a natural iterative decomposition structure for this retrieval process.
On codec restoration tasks for both speech and music, geometric iterative retrieval improves over single-shot token prediction and one-step regression baselines.

---

*Auto-collected on 2026-08-21.*