Paper Overview
Field: NLP/AI Authors: Daiwei Chen, Zhoutong Fu, Chengming Jiang Published: 2026-04-02 arXiv: 2604.02324
Summary
Language models (LMs) are increasingly extended with new learnable vocabulary tokens for domain-specific tasks, such as Semantic-ID tokens in generative recommendation. The standard practice initializes these new tokens as the mean of existing vocabulary embeddings, then relies on supervised fine-tuning to learn their representations. The authors present a systematic analysis of this strategy: through spectral and geometric diagnostics, they show that mean initialization collapses all new tokens into a degenerate subspace, erasing inter-token distinctions that subsequent fine-tuning struggles to fully recover. These findings suggest that token initialization is a key bottleneck when extending LMs with new vocabularies.
The GTI Hypothesis
Motivated by this diagnosis, the paper proposes the Grounded Token Initialization Hypothesis: linguistically grounding novel tokens in the pretrained embedding space before fine-tuning better enables the model to leverage its general-purpose knowledge for novel-token domains.
The hypothesis is operationalized as GTI (Grounded Token Initialization), a lightweight grounding stage that, prior to fine-tuning, maps new tokens to distinct, semantically meaningful locations in the pretrained embedding space using only paired linguistic supervision.
Results
- Despite its simplicity, GTI outperforms both mean initialization and existing auxiliary-task adaptation methods in the majority of evaluation settings across multiple generative recommendation benchmarks, including industry-scale and public datasets.
- Further analyses show that grounded embeddings produce richer inter-token structure that persists through fine-tuning.
- These results corroborate the hypothesis that initialization quality is a key bottleneck in vocabulary extension.
*Source: arXiv:2604.02324*