Have you ever wondered why virtual avatars always seem slightly out of sync between lip movements and voice? Traditional speech-driven facial animation takes a roundabout path: synthesize speech from text, extract features from that speech, then drive a face model with those features. Every step loses information and adds latency.
But what if speech and facial expression were fundamentally the same thing? When you say "ah," your mouth opening and vocal cord vibration happen simultaneously—they share the same neural command. So why not let AI generate sound and expression from the same "command" as well?
A research team from UNICAMP (Brazil) and Grenoble (France) did exactly that: they made speech and facial animation share one set of discrete tokens, with a single token stream decoding into both audio and 3D facial motion, achieving true audio-visual synchronization.
Four Speech Representations: Which Best Drives Faces?
The core question: what kind of speech representation is best suited for driving facial animation? The researchers compared four families of speech encoders:
1. HuBERT (semantic representation): a self-supervised model trained by predicting masked speech segments. It excels at capturing "what sound this is"—phoneme class information. This is the mainstream choice for facial animation today.
2. SpeechTokenizer (semantic + acoustic hybrid): multi-layer residual vector quantization, where the first layer distills HuBERT's semantic information and later layers encode acoustic details. Essentially "phonemes + prosody" combined.
3. WavTokenizer (acoustic representation): extreme compression, squeezing entire utterances into very few tokens via a single codebook. It optimizes for acoustic reconstruction quality—"sounds right"—without caring about "what was said."
4. CosyVoice2 (label representation): a generative model trained on ASR objectives; its token space is driven by "supervised labels"—each token corresponds to a distinct speech unit.
These four encoders were paired with two face decoders (GRU and Transformer), forming 8 combinations benchmarked on the VOCASET dataset.
Key Finding: Phonemic Information Is the Soul
The results are strikingly clear:
Semantic and label representations tie. HuBERT and CosyVoice2 perform comparably on facial animation quality, despite completely different training objectives—one self-supervised guessing, one supervised labeling. This shows the key to driving facial animation is not "how it's trained" but "what it encodes."
Phonemic information is a necessary condition. Probing analysis reveals a key pattern: as long as the speech representation encodes phoneme class information, facial animation works well; if it has only acoustic information lacking phonemic structure (like WavTokenizer), animation quality drops noticeably. Makes sense—mouth shape is mainly determined by "which sound was spoken," not "how the audio sounds."
Acoustic representations actually hurt. WavTokenizer can reconstruct speech perfectly, yet performs worst at facial animation. It compresses too aggressively, discarding phonemic structure. It's like someone who memorizes melodies but not lyrics—they can hum the tune but can't say the words, so naturally they can't form the right mouth shapes.
Hybrid representations fall short of pure semantics. SpeechTokenizer mixes semantic and acoustic information, but its facial animation quality trails pure-semantic HuBERT. The extra acoustic information becomes noise—facial animation doesn't need to know timbre or loudness, only how the mouth should move.
AVTTS: One Token Stream, Two Worlds
The most interesting part is the AVTTS (Audio Visual Text-to-Speech) pipeline.
Traditional speech-driven facial animation is two-stage: text → speech → facial animation. AVTTS collapses it into one stage: text → shared tokens → (speech + facial animation) output in parallel.
Specifically, the CosyVoice2 TTS model predicts a speech token sequence from text and a reference audio. The same token sequence is then fed into two decoders simultaneously: a flow-matching speech decoder generates the waveform, and a Transformer face decoder generates blendshape sequences. Because both decoders read the same tokens, speech and facial animation are synchronized by construction—no extra alignment mechanism needed.
It's like a person speaking: the brain issues one command, and the mouth and vocal cords execute it simultaneously. No "speak first, lip-sync later"—the command itself is synchronized.
The Bigger Picture
This work matters beyond "better facial animation." It points to a more fundamental direction: in multimodal speech-language models, different decoders can be attached to the same frozen LLM backbone. Speech decoders, face decoders, even emotion decoders can all start from the same shared tokens.
This means future virtual humans may no longer need "speech-driven face" as a separate module—the tokens generated by an LLM are themselves universal multimodal instructions, and different outputs are simply different executions of the same command.
Paper: https://arxiv.org/abs/2606.13630 Code: https://github.com/ProdCor/Token-to-Face