Grounded Token Initialization for New Vocabulary in Language Models

Paper Overview

• Field: NLP
• Authors: Daiwei Chen, Zhoutong Fu, Chengming Jiang
• Published: 2025-04-01
• arXiv: 2504.01260

Summary

Language models (LMs) are increasingly extended with new learnable vocabulary tokens for domain-specific tasks, such as Semantic-ID tokens in generative recommendation. The standard practice initializes these new tokens as the mean of existing vocabulary embeddings, then relies on supervised fine-tuning to learn their representations.

Key Findings

• Through spectral and geometric diagnostics, the authors show that mean initialization collapses all new tokens into a degenerate subspace, erasing inter-token distinctions that subsequent fine-tuning struggles to fully recover.
• These findings suggest that token initialization is a key bottleneck when extending LMs with new vocabularies.
• Motivated by this diagnosis, the authors propose the Grounded Token Initialization (GTI) Hypothesis: linguistically anchoring new tokens in the pretrained embedding space before fine-tuning better enables the model to leverage its general knowledge for the new token domain.
• GTI is implemented as a lightweight anchoring stage that maps new tokens to distinct, semantically meaningful positions in the pretrained embedding space using only pairwise linguistic supervision before fine-tuning.

Results

• Despite its simplicity, GTI outperforms mean initialization and existing auxiliary-task adaptation methods across most evaluation settings on multiple generative recommendation benchmarks, including both industrial-scale and public datasets.
• Further analysis shows that grounded embeddings produce richer inter-token structure that persists throughout fine-tuning, confirming the hypothesis that initialization quality is a key bottleneck in vocabulary extension.

*Auto-collected on 2026-04-04*