Your Voice Cloning System Is Secretly a Voice Anonymizer: XTTSv2 Repurposed for Voice Privacy

Forum topic · ✨步子哥 · 2026-08-29

Summary

Researchers at Bern University of Applied Sciences discovered that XTTSv2, an open-source voice cloning model by Coqui AI, works as a highly effective voice anonymization system with no retraining. Instead of cloning a speaker's own voice, they swap in a pseudo-speaker reference sample and iteratively refine the selection to minimize similarity to the original speaker. On CommonVoice across seven European languages, the method achieves an EER of 0.49 (near the theoretical maximum of 0.50), a WER of 0.16 (38% lower than baselines like SALT), and improves perceptual quality by +0.17 UTMOS over original recordings. The result shows that a system designed to copy identities can perfectly hide them, since cloning and replacing voices are technically the same operation. This has implications for both voice privacy engineering and policy: the most effective anonymization tools are also the voice cloning systems that privacy regulations may seek to restrict. Limitations include untested robustness against informed attackers, coverage of only European languages, and validation on a single architecture.

An Accidental Discovery

In August 2026, researchers at Bern University of Applied Sciences hit an engineering bottleneck in a voice privacy project: they needed to anonymize speakers' voices so that automatic speech recognition could still understand the content, but voice-print identification could not determine who was speaking.

The standard approach is to train dedicated anonymization models, but these systems share a common flaw: anonymized speech quality drops noticeably, sounding robotic or noisy.

Then they noticed something odd: an open-source voice cloning system called XTTSv2 can listen to a few seconds of your voice and generate synthetic speech that perfectly mimics your timbre. Its design goal is to copy identity.

The researchers used it in reverse: use XTTSv2 to generate a "pseudo-speaker's" voice, then replace the original speaker's voice with it. The result was surprising—not only was anonymization near the theoretical optimum, the speech quality was actually better than the original recordings.

A system that can perfectly copy your voice can also perfectly hide it.

The paper is arXiv:2608.27360, "Your Voice Cloning System is Secretly a Voice Anonymizer."

The Voice Anonymization Dilemma

A speech signal carries three layers of information:

1. Linguistic content: what is said 2. Prosody: intonation, rhythm, emotion 3. Biometric features: the voice print that uniquely identifies a speaker

Anonymization aims to erase the third layer while keeping the first two—like blurring a face in a photo while preserving the clothing and background.

Two core metrics measure effectiveness:

EER (Equal Error Rate): the error rate of an automatic speaker verification attack. Higher EER means harder identification; the theoretical maximum is 0.50, meaning attackers can only guess.
WER (Word Error Rate): lower WER means more intelligible content.

A good anonymizer pushes EER toward 0.50 while keeping WER low—a privacy-utility trade-off.

Problems with existing methods:

Signal processing approaches (McAdams coefficients, phase vocoders): no training needed, but weak protection that stronger attackers can defeat.
Neural approaches (SALT, MultiLingual): better privacy, but degraded quality. SALT reaches EER 0.37 on CommonVoice, but WER is 0.26 and UTMOS drops by 0.74.

XTTSv2: A "Reverse Weapon"

XTTSv2 is an open-source voice cloning system from Coqui AI. Given a 3-second sample, it generates speech that sounds like you, in 17 languages. Its architecture has three parts:

1. Text encoder: converts input text into linguistic tokens 2. Speaker encoder: extracts speaker features (voice print) from reference audio 3. Acoustic decoder: fuses tokens and speaker features into final speech

Normal usage: your text + your voice sample → synthetic speech as you.

The researchers' change: your text + someone else's voice sample (a pseudo-speaker) → synthetic speech saying your content in another voice.

That simple. No retraining, no model modification—just a different input.

One technical detail: how to choose the pseudo-speaker? A random pick may synthesize poorly (short/low-quality reference) or remain too similar to the original voice.

Iterative Refinement: Finding the Least Similar "Double"

1. Initial selection: randomly pick a reference sample from the pseudo-speaker pool 2. Synthesize: anonymize the utterance with XTTSv2 3. Evaluate: compute speaker similarity between synthetic and original speech 4. Adjust: if similarity is too high, swap the reference and re-synthesize 5. Iterate until similarity drops below a threshold or iterations are exhausted

The pool was built from multilingual, gender-balanced samples in public VoicePrivacy challenge datasets.

The elegance: rather than destroying the original voice print, the method replaces it with a carefully chosen double. The greater the difference, the better the anonymization. And as a cloning system, XTTSv2 naturally produces clean, natural synthetic speech—so quality doesn't suffer.

The Numbers

Experiments covered seven European languages (English, German, French, Spanish, Italian, Dutch, Portuguese) using CommonVoice and Multilingual LibriSpeech.

Core results (CommonVoice, vs. SALT and MultiLingual baselines):

| Metric | SALT | MultiLingual | XTTSv2 (this paper) | |--------|------|--------------|---------------------| | EER (privacy) | 0.37 | 0.46 | 0.49 | | WER (utility) | 0.26 | 0.27 | 0.16 | | ΔUTMOS (quality) | -0.74 | -0.62 | +0.17 |

Privacy: EER 0.49, near the 0.50 theoretical maximum—attackers can barely identify the original speaker.
Utility: WER 0.16, 38% lower than baselines.
Quality: ΔUTMOS +0.17, better than the original recordings, because XTTSv2's output is cleaner (no environmental noise).

On Multilingual LibriSpeech: EER 0.46, WER 0.16. Across all seven languages, EER stayed between 0.48 and 0.50.

Why This Matters

The deeper lesson: a system's design purpose and its actual capabilities can be far apart.

XTTSv2 was designed to clone voices, but its real capability is speech-to-speech mapping—given text and reference audio, generate fused synthetic speech. That capability is neutral. Swap "original speaker" for "pseudo-speaker" as the reference, and the same system turns from a cloner into an anonymizer.

This echoes a classic security principle: a tool's danger lies not in its purpose but in its capability boundary. Copying and replacing a voice are technically the same operation—only the reference sample differs.

Practically, the value is huge: no new models, no new data, no architecture changes—just a different input. Existing voice cloning systems (XTTSv2, VALL-E, Voicebox, etc.) can be used as anonymizers directly.

It also raises a policy question: if voice cloning systems are inherently anonymization systems, then "restricting voice cloning" and "protecting voice privacy" are partly contradictory—limiting cloning systems also limits the most effective anonymization tools.

The paper's acknowledged limitations:

Adversarial robustness: not tested against informed attackers who know the system details.
Language coverage: only European languages; tonal languages like Chinese and Japanese are untested.
Generalization: only XTTSv2 was tested; other cloning architectures remain unverified.

Paper: https://arxiv.org/abs/2608.27360 HTML version: https://arxiv.org/html/2608.27360v1 Related code:

Coqui TTS (XTTSv2): https://github.com/rm00cr/coqui-tts
Speaker Anonymization: https://github.com/DigitalPhonetics/speaker-anonymization
SALT baseline: https://github.com/BakerBunker/SALT