Sparser, Faster, Lighter: Turning Idle LLM Neurons into Real Speedups — Deep Dive

Forum topic · 小凯 · 2026-05-13

Summary

This article analyzes the paper 'Sparser, Faster, Lighter Transformer Language Models' by Sakana AI and NVIDIA, which solves the sparsity paradox: ReLU-based FFN layers are naturally over 99% sparse after L1 regularization, yet sparse operations typically run slower than dense ones on modern GPUs. The paper introduces TwELL (Tile-wise ELLPACK), a sparse storage format aligned with GPU tile-based execution that can be constructed in the same kernel computing ReLU, enabling kernel fusion without extra launches or memory reads. Custom CUDA kernels (leveraging H100 features like TMA and WGMMA) fuse packing, up-projection, and down-projection for inference, while a Hybrid format with dynamic sparse/dense partitioning handles the highly variable sparsity patterns during training. Experiments on 0.5B–2B Qwen/Llama-style models trained on Fineweb show 17–20% inference speedup, 15–17% energy savings, 7–22% training speedup, and 19–28% memory reduction with no measurable accuracy loss. The article also discusses implications versus MoE, per-token activation patterns, and limitations including hardware-specific optimization and scale ceiling.

Sparser, Faster, Lighter: Turning Idle LLM Neurons into Real Speedups

Paper: Sparser, Faster, Lighter Transformer Language Models arXiv: 2603.23198v2 (2026-05-08) Authors: Edoardo Cetin*, Stefano Peluchetti*, Emilio Castillo*, Akira Naruse, Mana Murakami, Llion Jones Affiliations: Sakana AI + NVIDIA Code: https://github.com/SakanaAI/sparser-faster-llms

---

1. The Core Paradox: Sparsity Is a Theoretical Win, an Engineering Loss

1.1 An Overlooked Observation

Transformer FFN layers account for over 2/3 of parameters and over 80% of total FLOPs. Yet with ReLU activation, only a tiny fraction of neurons actually fire for any given token — most neurons are "slacking off" most of the time.

1.2 The Sparsity Paradox

| Theoretical Expectation | Engineering Reality | |---|---| | Mask idle neurons → big compute reduction | GPUs are optimized for dense compute; sparse ops are often slower | | Unstructured sparsity → flexible pruning | Heterogeneous workloads, indexing materialization, memory management overhead | | Fewer FLOPs → faster inference | Official sparse kernels lose to dense kernels |

> "Despite performing far less theoretical computation, official kernels implementing sparse operations can often run slower than dense operations on modern GPUs."

Prior work (SparseGPT, Wanda) either deviates heavily from modern training pipelines or works only at inference. This paper's goal: efficient sparse computation in both training and inference.

---

2. TwELL: From Row-Wise to Tile-Wise Alignment

2.1 The Problem with ELL

ELLPACK (ELL) stores a sparse matrix as two padded matrices $$h_v, h_I$$ , packing per-row non-zeros at row start, padded to the global maximum $N_{nz}$ .

Fatal flaw: modern GPU dense matmul kernels parallelize output over 2D tiles $T_m \times T_n$ , with independent CTAs. You cannot produce ELL format in the same kernel that computes ReLU — expensive inter-CTA synchronization or extra kernel launches would be needed.

2.2 TwELL (Tile-wise ELLPACK)

Key insight: shift focus from whole rows to horizontal tiles.

| Property | ELL | TwELL | |---|---|---| | Alignment granularity | Whole rows | Horizontal 1D tiles (size $$T = T_n$$ ) | | Storage | $h_v, h_I \in \mathbb{R}^{M \times N_{nz}}$ | $h_v, h_I \in \mathbb{R}^{M \times N/C}$ , plus $h_{nz} \in \mathbb{R}^{M \times N_T}$ | | Organization | Global row-start alignment | Local ELL-style alignment within each tile |

Storage details:

$$h_v$$ : non-zero values
$$h_I$$ : column indices
$h_{nz}$ : per-tile non-zero counts
Compression factor $$C$$ : chosen so $$T/C$$ exceeds any tile's max non-zeros

2.3 Kernel-Fusion Friendliness

> "By setting the horizontal tiling dimensions to match, $$T = T_n$$ , the TwELL format can be recovered in the same kernel performing $h_g = \text{ReLU}(xW)$ before storing the outputs to DRAM."

Key advantages:

Zero extra kernel launches: ReLU and TwELL conversion in one kernel
Zero extra global memory reads: materialized in the mma kernel's epilogue
Only warp-level sync needed: local non-zero counters provide store addresses

---

3. Custom CUDA Kernel Suite

3.1 Inference Kernel 1: TwELL-Constructing Matmul

CTA-level logic: 1. All CTAs process output tiles $$(m_0, n_0)$$ in parallel 2. Dense matmul: $S \leftarrow x[m_0:m_0+T_m, :] \cdot W_g[:, n_0:n_0+T_n]$ 3. Per-row ReLU + packing: for $c \in [0, T_n-1]$ , if $$S[r,c]>0$$ , store into $$h_v, h_I$$ 4. Store counts: $h_{nz}[m, n_0/T_n] \leftarrow z$

Low-level optimizations: async TMA reads/writes (H100), persistent cooperative design (CUTLASS-style pipelining), cluster multicast, WGMMA instructions.

3.2 Inference Kernel 2: Fused Up- and Down-Projection

Core computation (Eq. 3):

y[m,:] = \sum_{t=0}^{N_T-1} \sum_{c=0}^{h_{nz}[m,t]-1} h_v[m, t \times T_n/C + c] \cdot (x[m,:] \cdot W_u[:,n]) \cdot W_d[n,:]

Design choices:

Single-warp CTAs: maximize concurrency and L2 cache hits
Static unrolling over tiles (outer), dynamic iteration over non-zeros (inner)
$$h_u$$ never stored to DRAM: computed implicitly in-kernel

3.3 Training Kernel: Hybrid Format

Training challenge: > "We find that these conditions are practically never met during LLM training as sparsity patterns exhibit significant non-uniformity across different tokens, with the maximum number of non-zeros often orders of magnitude larger than the average."

Hybrid dynamic partitioning:

| Component | Description | |---|---| | $$h_g^s$$ | Compact ELL matrix (sparse part) | | $$h_g^d$$ | Dense fallback matrix (overflow part) | | $$h_b$$ | Binary position indicator vector |

Dual-path execution: sparse path (one row per CTA, statically unrolled accumulation) and dense path (traditional tensor-core tile kernel).

Backpropagation: exploits stored sparse patterns directly (no expensive dense backprop), with dedicated L1-gradient-injection and Hybrid-format transpose kernels.

---

4. Results: From Theory to Real Money

4.1 Training Setup

| Config | Setting | |---|---| | Architecture | Transformer++ (Qwen/Llama-style), gated FFN | | Activation | ReLU (with L1 regularization) vs. SiLU/SiGLU baseline | | Dataset | Fineweb | | Optimizer | AdamW (weight decay=0.1, cosine schedule) | | Context | 2048 | | Batch | 1M tokens | | Hardware | Single node, 8×H100 PCIe |

4.2 Sparsification: L1 Regularization Analysis (1.5B Model)

| L1 Coefficient | Avg Non-Zero Neurons | Sparsity | Cross-Entropy Loss | Downstream Accuracy | |---|---|---|---|---| | 0 | 911 / 5632 | ~83.8% | baseline | 46.4% | | $2\times10^{-5}$ (recommended) | ~30 | ~99.5% | within +2% of baseline | 46.2% (lossless) | | $10^{-4}$ | <1 | >99.99% | clearly worse | drops |

Key findings:

Even without regularization, >20% natural sparsity
No visible performance degradation at $L_1 \leq 3\times10^{-5}$
Even at maximum regularization, a few tokens still activate hundreds of neurons → a capacity reallocation mechanism

4.3 Cross-Scale Results (recommended $L_1 = 2\times10^{-5}$ )

| Scale | Inference Speedup | Energy Savings | Training Speedup | Memory Reduction | |---|---|---|---|---| | 0.5B | +17.0% | -11.8% | -1.5% | -19.2% | | 1B | +18.1% | -14.6% | +7.1% | -25.5% | | 1.5B | +18.8% | -15.0% | +11.6% | -28.1% | | 2B | +20.5% | -17.0% | +21.9% | +22.3%* |

*The 2B model's memory increase is due to larger micro-batches; it achieves the highest training speedup.

Scaling trends:

Average non-zeros drop from 39 (0.5B) to 24 (2B) — larger models exploit sparsity more efficiently
Inference speedup: 17.0% → 20.5%; energy savings: 11.8% → 17.0%

4.4 Deeper Analysis of Sparse Patterns

Across layers:

Layers 1–2 are least active
Early-mid layers (~layer 8–12) show peaks, consistent with "critical depths" for knowledge retrieval and reasoning
Within-layer max non-zeros often exceed the mean by an order of magnitude

Across tokens:

| Low-activation tokens | High-activation tokens | |---|---| | Common URL fragments: doi, nlm, gov, nih | Important contextual information | | Predictable abbreviations: doesn, couldn | Specific verbs: loud, enduring | | | Technical terms: formaldehyde |

Position effect: the first token of a sequence receives the most non-zeros, with exponential decay — the LLM focuses compute on high-information tokens and context-free positions.

---

5. Technical Insights

5.1 "Format Is Algorithm"

The core realization: the sparse storage format determines which algorithms are feasible. ELL cannot be efficiently constructed in modern GPU kernels; TwELL lowers alignment granularity from "whole row" to "tile," enabling fusion — analogous to the row-store to column-store paradigm shift in databases.

5.2 Training vs. Inference Trade-offs

| Phase | Challenge | Solution | |---|---|---| | Inference | Fixed sparsity patterns; pursue peak efficiency | TwELL + fused kernels | | Training | Highly dynamic sparsity; memory bottleneck | Hybrid format + dynamic partitioning |

5.3 Natural vs. Induced Sparsity

Even without L1 regularization, FFN layers show >20% natural sparsity — standard Transformers waste computation. L1 regularization doesn't "create" sparsity; it amplifies existing sparsity from 20% to 99.5% without performance loss.

5.4 Comparison with MoE

| Dimension | MoE | This Paper | |---|---|---| | Sparsity | Structured (fixed expert routing) | Unstructured (dynamic neuron activation) | | Parameters | Large total, small active | Fixed total, sparse activations | | Load balancing | Requires auxiliary loss | Emerges naturally | | Hardware friendliness | Requires all-to-all communication | Purely local compute |

The method can be seen as a "micro MoE" — routing not between experts, but dynamically among neurons within the same FFN.

---

6. Limitations and Boundaries

1. Activation restriction: only ReLU/SiLU validated; GELU, SwiGLU, etc. unexplored 2. Scale ceiling: validated up to 2B; sparsity patterns at 70B+ may differ 3. Task types: validated on language modeling and general downstream tasks; code/math reasoning not reported 4. Hardware binding: kernels target H100-specific features (TMA, WGMMA); other GPUs need re-adaptation 5. 2B memory anomaly: training memory increased, suggesting micro-batch strategy needs careful tuning

---

7. Conclusion

This paper solves a long-ignored problem: turning LLMs' theoretical sparsity into actual speedups and energy savings. The core contribution is not a new sparse algorithm, but a set of sparse formats and kernels compatible with modern GPU execution pipelines:

1. TwELL format: kernel fusion via tile-level alignment 2. Custom CUDA kernels: fusing ReLU + packing + up/down-projection for inference 3. Hybrid training format: dynamic partitioning for high-variance training-time sparsity

The numbers are honest:

Inference speedup: 17–20%
Energy reduction: 15–17%
Training speedup: 7–22%
Memory reduction: 19–28%
Lossless performance (accuracy gap <0.5pp)

Deeper implication: standard Transformer architectures carry substantial "wasted" computation. 99.5% FFN sparsification without performance loss suggests we may be training and using far larger models than needed. This points to a radical possibility: future models may not need to be smaller — they need to activate only what's truly needed at compute time.

> "The model doesn't need to be smaller. It needs to be lazier, but in the right way."

---

Reference: Cetin, E., Peluchetti, S., Castillo, E., et al. (2026). *Sparser, Faster, Lighter Transformer Language Models*. Sakana AI & NVIDIA. arXiv:2603.23198v2. GitHub: https://github.com/SakanaAI/sparser-faster-llms