论文概要
Research area: NLP / Attention mechanisms Authors: Hee Seung Hwang, Sanghyuk Chun, Xindi Wu, Olga Russakovsky, Zhiwei Deng Published: 2026-10-27 arXiv: 2610.37645
Overview
Self-attention plays a key role in long-sequence modeling, but its quadratic time complexity fundamentally limits the scaling of Large Language Models. Multi-State Linear Attention (MSLA) mechanisms address this by splitting the context into segments and compressing each segment into its own memory state. However, MSLA aggregates information from each memory state using only the *mean* alignment between a query and each segment's keys, ignoring how the alignment is distributed within the segment.
Method: MERLIN
To model the relevance of a segment to a query more comprehensively, the authors propose MERLIN (MomEnt Routing for multi-state LINear attention), which tracks the first and second moments of each segment's keys. Using these moments, a query can recover both the mean and the variance of its alignment with each segment, enabling a more content-aware aggregation of per-state readouts.
Results
Experiments on multi-state variants of Gated DeltaNet-1.3B and Mamba-2-780M show that MERLIN consistently outperforms existing aggregation methods on long-context tasks:
- RULER needle-in-a-haystack and long-context QA: +8.3% (Gated DeltaNet-1.3B) and +7.0% (Mamba-2-780M)
- LongBench diverse subtasks: +13.5% and +12.0%
- MERLIN even outperforms baselines that consume 4x more memory
Original Abstract
> Self-attention plays a key role in long-sequence modeling, but its quadratic time complexity fundamentally limits the scaling of Large Language Models. Recently, Multi-State Linear Attention (MSLA) mechanisms have been proposed to make sequence modeling more efficient by splitting the context into segments and compressing each in its own memory state. MSLA aggregates the information extracted from each memory state using only the mean alignment between a query and each segment's keys, ignoring how the alignment is distributed within the segment. In order to model the relevance of a segment to a query more comprehensively and incorporate the variance of the alignment in the aggregation, we propose MERLIN (MomEnt Routing for multi-state LINear attention), a method that tracks the first and second moments of each segment's keys. Using the moments of keys, a query can recover both the mean and the variance of its alignment with each segment to perform a more content-aware aggregation of per-state readouts. Experiments on multi-state variants of Gated DeltaNet-1.3B and Mamba-2-780M show that MERLIN consistently outperforms existing aggregation methods on various long-context tasks. In particular, MERLIN outperforms baselines on RULER needle-in-a-haystack and long-context QA (+8.3% for Gated DeltaNet-1.3B and +7.0% for Mamba-2-780M) and diverse subtasks in LongBench (+13.5% and +12.0%). It even outperforms baselines that consume 4x more memory. MERLIN provides a simple and general routing mechanism for improving long-context capabilities of MSLA models.
Paper: arXiv:2610.37645