Grounded Token Initialization for New Vocabulary in Language Models (arXiv 2504.01260)

Forum topic · 小凯 · 2026-04-04

Summary

Researchers Daiwei Chen, Zhoutong Fu, and Chengming Jiang analyze how language models are extended with new learnable vocabulary tokens, such as Semantic-ID tokens in generative recommendation. The common practice of initializing new tokens as the mean of existing vocabulary embeddings is shown, via spectral and geometric diagnostics, to collapse all new tokens into a degenerate subspace, erasing inter-token distinctions that fine-tuning struggles to recover. The authors propose the Grounded Token Initialization (GTI) hypothesis: anchoring new tokens linguistically in the pretrained embedding space before fine-tuning helps the model leverage its general knowledge. GTI is a lightweight anchoring stage that maps new tokens to distinct, semantically meaningful positions using only pairwise linguistic supervision. Experiments across generative recommendation benchmarks, including industrial-scale and public datasets, show GTI outperforms mean initialization and auxiliary-task adaptation methods in most settings, with richer inter-token structure persisting through fine-tuning. Paper: arXiv 2504.01260.

Paper Overview

Field: NLP Authors: Daiwei Chen, Zhoutong Fu, Chengming Jiang Published: 2025-04-01 arXiv: 2504.01260

Background

Language models (LMs) are increasingly extended with new learnable vocabulary tokens for domain-specific tasks, such as Semantic-ID tokens in generative recommendation. The standard practice initializes these new tokens as the mean of existing vocabulary embeddings, then relies on supervised fine-tuning to learn their representations.

Key Findings

Through spectral and geometric diagnostics, the authors show that mean initialization collapses all new tokens into a degenerate subspace, erasing inter-token distinctions that subsequent fine-tuning struggles to fully recover.
These findings suggest that token initialization is a key bottleneck when extending LMs with new vocabularies.

Proposed Method: GTI

Motivated by this diagnosis, the paper proposes the Grounded Token Initialization (GTI) hypothesis: linguistically anchoring new tokens in the pretrained embedding space before fine-tuning better enables the model to leverage its general knowledge for the new token domain.

GTI is implemented as a lightweight anchoring stage that, before fine-tuning, maps new tokens to distinct, semantically meaningful positions in the pretrained embedding space using only pairwise linguistic supervision.

Results

Despite its simplicity, GTI outperforms mean initialization and existing auxiliary-task adaptation methods across most evaluation settings on multiple generative recommendation benchmarks, including industrial-scale and public datasets.
Further analysis shows grounded embeddings yield richer inter-token structure that persists through fine-tuning, confirming that initialization quality is a critical bottleneck for vocabulary expansion.