Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation for LLM Training

Forum topic · 小凯 · 2026-05-14

Summary

Pion is a novel spectrum-preserving optimizer for large language model (LLM) training, introduced by Kexuan Shi, Hanxuan Li, Zeju Qiu, Yandong Wen, Simon Buchholz, and Weiyang Liu (arXiv:2605.12492). Unlike additive optimizers such as Adam and Muon, Pion updates each weight matrix through left and right orthogonal transformations, preserving its singular values throughout training. This yields an optimization mechanism that modulates the geometry of weight matrices while keeping their spectral norm fixed. The authors derive the Pion update rule, systematically examine its design choices, and analyze its convergence behavior along with several key properties. Empirical results demonstrate that Pion offers a stable and competitive alternative to standard optimizers for both LLM pretraining and finetuning. The paper is relevant to researchers in machine learning, deep learning optimization, and LLM training efficiency.

Paper Overview

Research Area: Machine Learning (ML)
Authors: Kexuan Shi, Hanxuan Li, Zeju Qiu, Yandong Wen, Simon Buchholz, Weiyang Liu
Published: 2026-05-12
arXiv: 2605.12492

Abstract

We introduce Pion, a spectrum-preserving optimizer for large language model (LLM) training based on orthogonal equivalence transformation. Unlike additive optimizers such as Adam and Muon, Pion updates each weight matrix through left and right orthogonal transformations, preserving its singular values throughout training. This yields an optimization mechanism that modulates the geometry of weight matrices while keeping their spectral norm fixed.

The paper derives the Pion update rule, systematically examines its design choices, and analyzes its convergence behavior along with several key properties. Empirical results show that Pion offers a stable and competitive alternative to standard optimizers for both LLM pretraining and finetuning.

Key Takeaways

Pion is an orthogonal-equivalence-based optimizer, distinct from additive optimizers like Adam and Muon.
It preserves the singular values of each weight matrix across the entire training process.
The mechanism modulates matrix geometry while keeping spectral norms fixed.
The paper includes convergence analysis and a systematic study of design choices.
Experiments cover both LLM pretraining and finetuning, showing stable, competitive performance.

---

*Auto-collected on 2026-05-14.*