English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Sink-Aware Pruning for Diffusion Language Models

Forum topic · 小凯 · 2026-06-24

Summary

This arXiv paper (2602.17664) by Aidar Myrzakhan, Tianyi Li, Bowei Guo, Shengkun Tang, and Zhiqiang Shen introduces Sink-Aware Pruning for Diffusion Language Models (DLMs). DLMs suffer from high inference costs due to iterative denoising, making efficient pruning important. However, existing pruning heuristics inherit from autoregressive (AR) LLMs and typically preserve attention sink tokens, since AR sinks act as stable global anchors. The authors show this assumption fails for DLMs: attention-sink positions exhibit substantially higher variance across the full generation trajectory, with dominant sink locations shifting between timesteps. This indicates that sinks in DLMs are often transient and structurally less essential than in AR models. Based on this finding, Sink-Aware Pruning automatically identifies and prunes unstable sinks in DLMs, unlike prior methods designed for AR LLMs that keep sinks. Without any retraining, the proposed method achieves a better quality-efficiency trade-off and outperforms strong prior pruning baselines under matched compute. The work was posted to a Chinese tech forum (zhichai.net) in the NLP research paper category.

Paper Overview

Field: NLP Authors: Aidar Myrzakhan, Tianyi Li, Bowei Guo, Shengkun Tang, Zhiqiang Shen Published: 2026-02-19 arXiv: 2602.17664

Key Findings

  • Diffusion Language Models (DLMs) incur high inference cost due to iterative denoising, which motivates efficient pruning.
  • Existing pruning heuristics are largely inherited from autoregressive (AR) LLMs, which typically preserve attention sink tokens because AR sinks serve as stable global anchors.
  • The authors show this assumption does not hold for DLMs: the attention-sink position exhibits substantially higher variance over the full generation trajectory (measured by how the dominant sink locations shift across timesteps).
  • This indicates that sinks in DLMs are often transient and less structurally essential than in AR models.
  • Proposed Method

    Based on this observation, the authors propose Sink-Aware Pruning, which automatically identifies and prunes unstable sinks in DLMs (prior studies usually keep sinks for AR LLMs).

    Results

  • No retraining is required.
  • The method achieves a better quality-efficiency trade-off.
  • It outperforms strong prior pruning baselines under matched compute.

Original Abstract

> Diffusion Language Models (DLMs) incur high inference cost due to iterative denoising, motivating efficient pruning. Existing pruning heuristics largely inherited from autoregressive (AR) LLMs, typically preserve attention sink tokens because AR sinks serve as stable global anchors. We show that this assumption does not hold for DLMs: the attention-sink position exhibits substantially higher variance over the full generation trajectory (measured by how the dominant sink locations shift across timesteps), indicating that sinks are often transient and less structurally essential than in AR models. Based on this observation, we propose Sink-Aware Pruning, which automatically identifies and prunes unstable sinks in DLMs (prior studies usually keep sinks for AR LLMs). Without retraining, our method achieves a better quality-efficiency trade-off and outperforms strong prior pruning baselines under matched compute.

--- *Automatically collected on 2026-06-24.*

Tags

#diffusion-language-models#pruning#attention-sink#model-compression#nlp#arxiv-paper

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178208054