Grounded Token Initialization: Better Vocabulary Extension for Generative Recommendation LMs

Forum topic · 小凯 · 2026-04-05

Summary

A new arXiv paper (2604.02324) by Daiwei Chen, Zhoutong Fu, and Chengming Jiang examines how language models are extended with new learnable vocabulary tokens, such as Semantic-ID tokens in generative recommendation. The authors show that the standard mean-of-embeddings initialization collapses all new tokens into a degenerate subspace, erasing inter-token distinctions that fine-tuning cannot fully recover—making initialization a key bottleneck in vocabulary extension. They propose the Grounded Token Initialization (GTI) hypothesis: linguistically grounding novel tokens in the pretrained embedding space before fine-tuning lets the model better leverage its general-purpose knowledge. GTI is a lightweight grounding stage that uses only paired linguistic supervision to map new tokens to distinct, semantically meaningful locations in the pretrained embedding space. Across multiple generative recommendation benchmarks, including industry-scale and public datasets, GTI outperforms mean initialization and existing auxiliary-task adaptation methods in most settings, and grounded embeddings retain richer inter-token structure through fine-tuning.

Paper Overview

Field: NLP/AI Authors: Daiwei Chen, Zhoutong Fu, Chengming Jiang Published: 2026-04-02 arXiv: 2604.02324

Problem

Language models are increasingly extended with new learnable vocabulary tokens for domain-specific tasks, such as Semantic-ID tokens in generative recommendation. The standard practice initializes these new tokens as the mean of existing vocabulary embeddings, then relies on supervised fine-tuning to learn their representations.

Through spectral and geometric diagnostics, the authors show that mean initialization collapses all new tokens into a degenerate subspace, erasing inter-token distinctions that subsequent fine-tuning struggles to fully recover. This suggests that token initialization is a key bottleneck when extending language models with new vocabularies.

Proposed Approach

Motivated by this diagnosis, the paper proposes the Grounded Token Initialization (GTI) Hypothesis: linguistically grounding novel tokens in the pretrained embedding space before fine-tuning better enables the model to leverage its general-purpose knowledge for novel-token domains.

GTI is a lightweight grounding stage that, prior to fine-tuning, maps new tokens to distinct, semantically meaningful locations in the pretrained embedding space using only paired linguistic supervision.

Results

Despite its simplicity, GTI outperforms both mean initialization and existing auxiliary-task adaptation methods in the majority of evaluation settings across multiple generative recommendation benchmarks, including industry-scale and public datasets.
Further analyses show that grounded embeddings produce richer inter-token structure that persists through fine-tuning.
The findings corroborate the hypothesis that initialization quality is a key bottleneck in vocabulary extension.

Original Abstract

> Language models (LMs) are increasingly extended with new learnable vocabulary tokens for domain-specific tasks, such as Semantic-ID tokens in generative recommendation. The standard practice initializes these new tokens as the mean of existing vocabulary embeddings, then relies on supervised fine-tuning to learn their representations. We present a systematic analysis of this strategy: through spectral and geometric diagnostics, we show that mean initialization collapses all new tokens into a degenerate subspace, erasing inter-token distinctions that subsequent fine-tuning struggles to fully recover. These findings suggest that token initialization is a key bottleneck when extending LMs with new vocabularies. Motivated by this diagnosis, we propose the Grounded Token Initialization Hypothesis: linguistically grounding novel tokens in the pretrained embedding space before fine-tuning better enables the model to leverage its general-purpose knowledge for novel-token domains. We operationalize this hypothesis as GTI (Grounded Token Initialization), a lightweight grounding stage that, prior to fine-tuning, maps new tokens to distinct, semantically meaningful locations in the pretrained embedding space using only paired linguistic supervision. Despite its simplicity, GTI outperforms both mean initialization and existing auxiliary-task adaptation methods in the majority of evaluation settings across multiple generative recommendation benchmarks, including industry-scale and public datasets. Further analyses show that grounded embeddings produce richer inter-token structure that persists through fine-tuning, corroborating the hypothesis that initialization quality is a key bottleneck in vocabulary extension.

--- *Auto-collected on 2026-04-05*