ScenA: Reference-Driven Multi-Speaker Audio Scene Generation with a Flow-Matching Foundation Model

Summary

ScenA is a method for generating multi-speaker conversational audio by conditioning a text-to-audio flow-matching foundation model, pretrained on large-scale in-the-wild data, directly on multiple reference voices and a free-form natural language scene prompt. Unlike prior dialogue systems that bind speakers to utterances via per-turn tags, multi-stream transcriptions, or learnable speaker embeddings, ScenA requires no per-turn structure. Reference latents are concatenated into the model's token sequence and distinguished by lightweight identity-aware positional encodings, enabling generation of natural audio with background noise, room acoustics, overlapping dialogue, and paralinguistic events. The authors identify a key failure mode called the Reference Shortcut: under standard noise schedules, the model can match references by acoustic similarity to the noisy target and bypass the text prompt. This is mitigated with a high-noise-biased timestep distribution that forces text-based speaker assignment. On the CoVoStress...