Have you ever wondered why a virtual streamer's lip movements always seem slightly out of sync with the voice? Traditional speech-driven facial animation takes a roundabout path: first synthesize speech from text, then extract features from the audio, and finally use those features to drive a face model. Every step loses information and adds latency.
But what if speech and facial expression were fundamentally the same thing? When you say "ah," your mouth opening and vocal cord vibration happen simultaneously—they share the same neural command. So why not let AI also generate sound and expression from a single shared "command"?
A research team from UNICAMP (Brazil) and Grenoble (France) did exactly that: they let speech and facial animation share one set of discrete tokens, with a single token stream decoding into both audio and 3D facial motion—achieving genuine audio-visual synchronization.
Four Speech Representations: Which Best Drives the Face?
The core question: which speech representation is best suited to drive facial animation? The researchers compared four classes of speech encoders:
1. HuBERT (semantic representation): a self-supervised model trained by predicting masked speech segments. It excels at capturing "what sound this is"—phonemic class information. This is the mainstream choice for current facial animation.
2. SpeechTokenizer (semantic + acoustic hybrid): multi-layer residual vector quantization where the first layer distills HuBERT's semantic information and later layers encode acoustic details—roughly "phonemes + intonation."
3. WavTokenizer (acoustic representation): extreme compression, squeezing entire utterances into very few tokens via a single codebook. It pursues acoustic reconstruction quality—"sounds like it"—without caring about "what was said."
4. CosyVoice2 (label representation): a generative model trained on ASR objectives; its token space is driven by "supervised labels," with each token corresponding to a clear speech unit.
These four encoders were each paired with two face decoders (GRU and Transformer), forming 8 combinations evaluated on the VOCASET dataset.
Key Finding: Phonemic Information Is the Soul
The results are strikingly clear:
Semantic and label representations tie. HuBERT and CosyVoice2 perform comparably on facial animation quality, despite completely different training objectives—one via self-supervised guessing, the other via supervised labels. This shows the key to driving facial animation is not "how it's trained" but "what is encoded."
Phonemic information is a necessary condition. Probing analysis reveals a crucial pattern: as long as the speech representation encodes phonemic class information, facial animation works well; if it contains only acoustic information without phonemic structure (like WavTokenizer), animation quality clearly degrades. This makes sense—mouth shape is determined mainly by "what sound was said," not "how it sounds."
Acoustic representations actually drag things down. WavTokenizer reconstructs speech perfectly, yet performs worst on facial animation. Its compression is too aggressive and discards phonemic structure—like someone who remembers only melodies but not lyrics: they can hum the tune, but can't say the words, and certainly can't produce the right mouth shapes.
Hybrid representations underperform pure semantics. SpeechTokenizer mixes semantic and acoustic information, but its facial animation results fall short of pure-semantic HuBERT. The extra acoustic information acts as noise—facial animation doesn't need to know timbre or loudness, only how the mouth should move.
AVTTS: One Token Stream, Two Worlds
The most interesting part is the AVTTS (Audio Visual Text-to-Speech) pipeline.
Traditional speech-driven facial animation is two-stage: text → speech → facial animation. AVTTS makes it single-stage: text → shared tokens → (speech + facial animation) in parallel.
Concretely, the CosyVoice2 TTS model predicts a speech token sequence from text and a reference audio. The same token sequence is then fed into two decoders simultaneously: a flow-matching speech decoder generates the waveform, and a Transformer face decoder generates blendshape sequences. Because both decoders read the same token stream, speech and facial animation are synchronized by construction—no additional alignment mechanism needed.
It's like how when a person speaks, the brain issues one command that the mouth and vocal cords execute simultaneously. There's no "speak first, lip-sync later"—the command itself is synchronized.
The Bigger Picture
This work's significance goes beyond "better facial animation." It points to a more fundamental direction: in multimodal speech-language models, different decoders can attach to the same frozen LLM backbone. Speech decoders, face decoders, even emotion decoders can all start from one shared token set.
This means future virtual humans may no longer need "speech-driven face" as a separate module—tokens generated by the LLM are themselves universal multimodal instructions, and various outputs are just different executions of the same command.
Paper: https://arxiv.org/abs/2606.13630 Code: https://github.com/ProdCor/Token-to-Face