ScenA: Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors

Forum topic · 小凯 · 2026-06-19

Summary

ScenA is a new method for generating multi-speaker conversational audio scenes, introduced by Michael Finkelson, Daniel Segal, and Eitan Richardson (arXiv:2506.14971). Instead of binding speakers to utterances via per-turn tags or speaker embeddings, ScenA conditions a text-to-audio flow-matching foundation model, pretrained on large-scale in-the-wild data, on multiple reference voices plus a free-form natural language prompt describing the whole scene. Reference latents are concatenated into the token sequence with lightweight identity-aware positional encodings, inheriting natural audio qualities like background noise, room acoustics, overlapping dialogue, and paralinguistic events. The authors identify a key failure mode, the Reference Shortcut: under standard noise schedules, the model matches references by acoustic similarity to the noisy target, bypassing the text. A high-noise-biased timestep distribution forces reliance on the text prompt for speaker assignment. On the CoVoMix2-Dialogue benchmark, ScenA outperforms existing multi-speaker systems on speaker-binding metrics while producing rich audio with overlapping speech, emotional vocalizations, and ambient sound.

Paper Overview

Field: CV / Audio Generation
Authors: Michael Finkelson, Daniel Segal, Eitan Richardson
Published: 2026-06-19
arXiv: 2506.14971

Background

Existing multi-speaker dialogue systems bind speakers to utterances through structured supervision: per-turn tags, multi-stream transcriptions, or learnable speaker embeddings. These speech-only pipelines produce clean vocal sequences but lack the ambient texture of real conversations.

Method: ScenA

ScenA takes a different approach by conditioning a text-to-audio flow-matching foundation model, pretrained on large-scale in-the-wild data, directly on:

Multiple reference voices
A free-form natural language prompt describing an entire multi-speaker audio scene

Leveraging the foundation model inherits its capacity for natural, non-studio audio: background noise, room acoustics, overlapping dialogue, and spontaneous paralinguistic events — with multi-speaker control and no per-turn structure.

Concretely, reference latents are concatenated into the model's token sequence and distinguished by lightweight identity-aware positional encodings.

Key Challenge: The Reference Shortcut

The authors identify a critical obstacle: during training under standard noise schedules, the model can identify the matching reference via acoustic similarity to the noisy target, bypassing the text prompt entirely.

Solution: a high-noise-biased timestep distribution that forces the model to rely on the text prompt for speaker assignment.

Results

Evaluated on the CoVoMix2-Dialogue benchmark, ScenA:

Outperforms existing multi-speaker systems on speaker-binding metrics
Generates rich conversational audio with overlapping speech, emotional vocalizations, and ambient sound

Conclusion

The results demonstrate the advantage of using a general-purpose audio model conditioned on a free-form scene description, rather than passing structured dialog scripts through a speech-only pipeline.

---

*Auto-collected on 2026-06-19*