An Accidental Discovery
In August 2026, researchers at Bern University of Applied Sciences hit an engineering bottleneck while working on a speech privacy project: they needed to anonymize speakers' voices—so that automatic speech recognition could still understand the content, but voice-print identification could not determine who was speaking.
The standard approach is training dedicated anonymization models, but these systems share a common flaw: noticeably degraded speech quality after anonymization, sounding robotic or noisy.
Then they noticed something odd: there is an open-source voice cloning system called XTTSv2 that can listen to a few seconds of your voice and generate synthetic speech perfectly mimicking your timbre. The system's design goal is to replicate identity—making the synthetic voice sound exactly like you.
But the researchers used it in reverse: generate a "fake" person's voice with XTTSv2, then replace the original speaker's voice with this fake voice. The result was unexpected—anonymization performance near theoretical optimum, and speech quality even better than the original recording.
A system that can perfectly clone your voice can also perfectly hide it.
The paper is arXiv:2608.27360, *Your Voice Cloning System is Secretly a Voice Anonymizer*.
The Speech Anonymization Dilemma
Speech signals carry three layers of information:
1. Linguistic content: what words are spoken 2. Prosody: intonation, rhythm, emotion 3. Biometric features: the voice print that uniquely identifies a speaker
Anonymization aims to erase the third layer while keeping the first two—like blurring a face in a photo while preserving clothing and background.
Two core metrics measure anonymization effectiveness:
- EER (Equal Error Rate): measures resistance against automatic speaker verification attacks. Higher EER means attackers find it harder to identify the speaker. The theoretical maximum is 0.50—meaning attackers cannot distinguish at all and can only guess.
- WER (Word Error Rate): speech recognition error rate. Lower WER means clearer, more intelligible content.
- Signal processing approaches (e.g., McAdams coefficients, phase vocoders): no training required, but limited privacy protection—slightly stronger attackers can break through.
- Neural network approaches (e.g., SALT, MultiLingual): require dedicated anonymization model training, better privacy, but noticeably degraded speech quality. SALT reaches EER 0.37 on CommonVoice, but WER climbs to 0.26, and the quality metric UTMOS drops by 0.74.
- Privacy: EER 0.49, near the theoretical maximum of 0.50. Attackers are almost completely unable to identify the original speaker.
- Usability: WER 0.16, 38% lower than baselines. Speech recognition understands the content more easily.
- Quality: ΔUTMOS +0.17—better than the original recordings. XTTSv2's synthetic speech is cleaner than raw recordings (no environmental noise).
- Adversarial robustness: against "informed attackers" who know the system details, anonymization effectiveness may drop. This scenario was not tested.
- Language coverage: only European languages were tested; effects on tonal languages like Chinese and Japanese are unknown.
- Generalization: only XTTSv2 was tested; whether other voice cloning architectures share this property remains to be verified.
- Coqui TTS (XTTSv2): https://github.com/rm00cr/coqui-tts
- Speaker Anonymization: https://github.com/DigitalPhonetics/speaker-anonymization
- SALT baseline: https://github.com/BakerBunker/SALT
A good anonymization system should push EER toward 0.50 while keeping WER low—a privacy-utility trade-off.
Problems with existing methods:
XTTSv2: A "Reverse Weapon"
XTTSv2 is an open-source voice cloning system developed by Coqui AI. Given 3 seconds of your voice, it generates synthetic speech that sounds like you, supporting 17 languages.
Its architecture has three parts:
1. Text encoder: converts input text into language tokens 2. Speaker encoder: extracts speaker features (voice print) from reference audio 3. Acoustic decoder: fuses language tokens with speaker features to generate the final speech
Normal usage: your text + your voice sample → synthetic speech of you speaking.
The researchers' modification: your text + someone else's voice sample (a pseudo-speaker) → synthetic speech speaking your content in someone else's voice.
That's it. No retraining, no model modification—just a different input.
One technical detail: how to choose the pseudo-speaker? A random pick may synthesize poorly (reference too short or low quality), or produce a voice too similar to the original (if the two voice prints are naturally close).
Iterative Refinement: Finding the Least Similar "Stand-In"
The researchers designed an iterative refinement pipeline:
1. Initial selection: randomly pick a reference sample from the pseudo-speaker pool 2. Synthesis: generate anonymized speech with XTTSv2 3. Evaluation: compute speaker similarity between synthetic and original speech 4. Adjustment: if similarity is too high (insufficient anonymization), swap in a different reference sample and re-synthesize 5. Iteration: repeat steps 2–4 until similarity falls below a threshold or maximum iterations are reached
The pseudo-speaker pool is carefully constructed: the researchers selected multilingual, multi-gender reference samples from public VoicePrivacy challenge datasets to cover diverse timbres.
The pipeline's elegance: it doesn't "destroy" the original voice print—it "replaces" it with a carefully chosen stand-in voice. The greater the difference between the stand-in and the original speaker, the better the anonymization.
And since XTTSv2 is a voice cloning system, it naturally excels at producing high-quality, natural-sounding synthetic speech—guaranteeing quality doesn't degrade after replacement.
The Numbers
Experiments covered 7 European languages (English, German, French, Spanish, Italian, Dutch, Portuguese) using CommonVoice and Multilingual LibriSpeech datasets.
Core results (CommonVoice, compared against SALT and MultiLingual baselines):
| Metric | SALT | MultiLingual | XTTSv2 (this work) | |--------|------|--------------|--------------------| | EER (privacy) | 0.37 | 0.46 | 0.49 | | WER (usability) | 0.26 | 0.27 | 0.16 | | ΔUTMOS (quality) | -0.74 | -0.62 | +0.17 |
Leading on all three metrics:
Results on Multilingual LibriSpeech are similar: EER 0.46, WER 0.16.
Cross-language performance is stable: across the 7 languages on CommonVoice, EER stays within 0.48–0.50, with virtually no variation.
Why This Matters
The paper's deeper insight: a system's design purpose and its actual capabilities can be vastly different.
XTTSv2 was designed to clone voices—its goal is making synthetic speech sound like the original speaker. But its actual capability is "speech-to-speech mapping": given text and reference audio, generate synthetic speech blending both.
That mapping capability is neutral. Swap the "reference audio" from "original speaker" to "pseudo-speaker," and the same system transforms from "cloner" to "anonymizer."
This recalls a classic security principle: a tool's danger lies not in its design purpose but in its capability boundary. A system that can perfectly replicate a voice is necessarily a system that can perfectly replace a voice—because "replication" and "replacement" are technically the same operation, differing only in reference sample selection.
From an engineering standpoint, the discovery is hugely valuable: no new model training, no new data collection, no architecture changes—just a different input. Existing voice cloning systems (XTTSv2, VALL-E, Voicebox, etc.) can be used directly as anonymization systems.
This also raises an interesting policy question: if voice cloning systems are inherently anonymization systems, then "restricting voice cloning" and "protecting voice privacy" are to some extent contradictory goals—restricting cloning systems also restricts the most effective anonymization tool.
The paper's stated limitations:
Paper link: https://arxiv.org/abs/2608.27360 HTML version: https://arxiv.org/html/2608.27360v1 Related code: