What Gets Unmasked First? Trajectory Analysis of Masked Diffusion Language Models for Graph-to-Text Generation

Forum topic · 小凯 · 2026-06-02

Summary

This paper presents the first systematic study of masked diffusion language models (MDLMs) for graph-to-text generation. By analyzing MDLM generation trajectories—the order in which tokens are unmasked during iterative decoding—the authors find that, unlike autoregressive LLMs that generate text linearly, MDLMs naturally prioritize entities first, followed by relational and function words, with structural tokens resolved last. The study also uncovers a previously undocumented failure mode of supervised fine-tuning (SFT): SFT disrupts this strategy by prematurely anchoring structural end-of-sequence tokens early in the decoding trajectory, effectively fixing output length and causing information omission or hallucination. To address this, the authors propose lambda-scaled structural decoding, a training-free inference-time modification that downweights structural token confidence, restoring up to +9.4 BLEU-4. Finally, they introduce Graph-LLaDA, which integrates a graph transformer encoder into the LLaDA decoding process to explicitly incorporate relational graph structure. Cross-dataset evaluation reveals that prior baselines overfit dataset-specific patterns, while LLM- and MDLM-based approaches generalize significantly better. Paper: arXiv 2605.31564, by Qing Wang, Jacob Devasier, and Chengkai Li.

Paper Overview

Field: NLP / AI

Authors: Qing Wang, Jacob Devasier, Chengkai Li

arXiv: 2605.31564

PDF: 2605.31564.pdf

Key Findings

This paper is the first systematic study of masked diffusion language models (MDLMs) for graph-to-text generation.

Generation Trajectories

By analyzing MDLM generation trajectories—the order in which tokens are unmasked during iterative decoding—the authors find that, unlike autoregressive LLMs that generate text linearly, MDLMs naturally adopt a content-first strategy:

1. Entities are unmasked first 2. Relational words and function words come next 3. Structural tokens are resolved last

An SFT Failure Mode

The paper identifies a previously undocumented failure mode of supervised fine-tuning (SFT): SFT disrupts this natural strategy by prematurely anchoring structural end-of-sequence tokens early in the decoding trajectory. This effectively fixes the output length, potentially leading to information omission or hallucination.

Lambda-Scaled Structural Decoding

To fix this, the authors propose lambda-scaled structural decoding, a training-free inference-time modification that reduces structural token confidence during decoding, restoring +9.4 BLEU-4.

Graph-LLaDA

Finally, the paper introduces Graph-LLaDA, which integrates a graph transformer encoder into the LLaDA decoding process to explicitly incorporate relational graph structure. Cross-dataset evaluation reveals that prior baselines overfit dataset-specific patterns, while LLM- and MDLM-based methods generalize significantly better.

---

*Auto-collected on 2026-06-02*