Toward Consistent World Models: LSE-MTP Combines Multi-Token Prediction with Latent Semantic Enhancement (arXiv 2504.06255)

Forum topic · 小凯 · 2026-04-09

Summary

This arXiv paper (2504.06255) by Qimin Zhong, Hao Liao, and Haiming Qin investigates whether Large Language Models develop coherent internal world models. The authors provide a theoretical analysis of the gradient inductive bias of Multi-Token Prediction (MTP), supported by empirical evidence, showing that MTP induces representational contractivity through gradient coupling and promotes convergence toward internal belief states. However, they identify a key limitation: standard MTP often suffers from structural hallucinations, where discrete token supervision encourages illegal shortcuts in latent space that violate environmental constraints. To address this, they propose Latent Semantic Enhancement MTP (LSE-MTP), which anchors predictions to ground-truth hidden-state trajectories. Experiments on synthetic graphs and real-world Manhattan taxi data demonstrate that LSE-MTP effectively bridges the gap between discrete tokens and continuous state representations, improves representation alignment, reduces structural hallucinations, and increases robustness to perturbations. Published April 8, 2025.

Paper Overview

Field: NLP Authors: Qimin Zhong, Hao Liao, Haiming Qin Published: 2025-04-08 arXiv: 2504.06255

Abstract

Whether Large Language Models (LLMs) develop coherent internal world models remains a core debate. While conventional Next-Token Prediction (NTP) focuses on one-step-ahead supervision, Multi-Token Prediction (MTP) has shown promise in learning more structured representations.

In this work, the authors provide a theoretical perspective analyzing the gradient inductive bias of MTP, supported by empirical evidence, showing that MTP promotes the convergence toward internal belief states by inducing representational contractivity via gradient coupling.

However, the paper reveals that standard MTP frequently suffers from structural hallucinations: discrete token supervision encourages illegal shortcuts in latent space that violate environmental constraints.

Proposed Method: LSE-MTP

To address this issue, the authors propose Latent Semantic Enhancement MTP (LSE-MTP), which anchors predictions to ground-truth hidden-state trajectories.

Results

Experiments on synthetic graphs and real-world Manhattan taxi data show that LSE-MTP:

Effectively bridges the gap between discrete tokens and continuous state representations
Enhances representation alignment
Reduces structural hallucinations
Improves robustness to perturbations

---

*Auto-collected on 2026-04-09*