HyScale: Scaling Long-Context Training for Learned Sparse Attention

Forum topic · 小凯 · 2026-10-26

Summary

HyScale is a training system for learned sparse attention, an architecture suited to million-token contexts that has seen production adoption since DeepSeek-V32. As sequence lengths grow, token selection becomes the training bottleneck: each query must score every previous token even though sparse attention reads only the top-k, and existing systems materialize full score tables while repeating selection during recomputation. HyScale's key insight is that selection output is a set of indices, not an activation. It introduces HyTopK, which finds each query's exact top-k on the fly without materializing a score table, and selection replay, which reuses forward-pass selection results during recomputation. Combined with holistic system optimizations, HyScale accelerates GLM-5.1 training on 1M-token sequences by 2.07x on 512 H20 GPUs versus Megatron-LM, and speeds up DeepSeek-V4-Flash CSA layers by 2.01x and Qwen3.8-Flash-Next QSA layers by 1.57x on 32 H800 GPUs. Paper: arXiv 2610.32057.

Overview

Field: Machine Learning Authors: Yangsheng Deng, Xinjing Huang, Yulong Wang, Yunzhuo Liu, Jun Hu, Shuyong Tang, Junchi Zeng, Mingji Han, Ray Ying, Key Zhang, Luo Mai Published: 2026-10-27 arXiv: 2610.32057

We present HyScale, a long-context training system for learned sparse attention. Learned sparse attention is a promising architecture for million-token contexts, widely adopted in production since DeepSeek-V3.2. Selection, however, grows into the training bottleneck: for each query, it scores every previous token whereas sparse attention reads only k; worse, current systems materialize a full score table and repeat selection during recomputation.

Key Idea

The output of selection is not an activation, but rather a set of indices. Building on this insight, HyScale introduces two techniques:

HyTopK: finds each query's exact top-k on the fly, without materializing a score table.
Selection replay: reuses forward-pass selection results during recomputation.

Together with holistic system optimizations addressing end-to-end training challenges, these techniques eliminate the selection bottleneck.

Results

2.07x speedup training GLM-5.1 on 1M-token sequences versus Megatron-LM, on 512 H20 GPUs.
2.01x speedup for DeepSeek-V4-Flash CSA layers at 1M tokens, on 32 H800 GPUs.
1.57x speedup for Qwen3.8-Flash-Next QSA layers at 1M tokens, on 32 H800 GPUs.

Paper: arXiv:2610.32057