Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformations for LLM Training

Forum topic · 小凯 · 2026-05-14

Summary

Pion is a spectrum-preserving optimizer for large language model (LLM) training, introduced by Kexuan Shi, Hanxuan Li, Zeju Qiu, Yandong Wen, Simon Buchholz, and Weiyang Liu (arXiv:2605.12492). Unlike additive optimizers such as Adam and Muon, Pion updates each weight matrix through left and right orthogonal transformations, preserving its singular values throughout training. This produces an optimization mechanism that modulates the geometry of weight matrices while keeping their spectral norm fixed. The paper derives the Pion update rule, systematically examines its design choices, and analyzes convergence behavior along with several key properties. Empirical results demonstrate that Pion offers a stable and competitive alternative to standard optimizers for both LLM pretraining and finetuning.

Overview

Research area: Machine Learning

Authors: Kexuan Shi, Hanxuan Li, Zeju Qiu, Yandong Wen, Simon Buchholz, Weiyang Liu

arXiv: 2605.12492

Abstract (translation)

We introduce Pion, a spectrum-preserving optimizer for large language model (LLM) training based on orthogonal equivalence transformation. Unlike additive optimizers such as Adam and Muon, Pion updates each weight matrix through left and right orthogonal transformations, preserving its singular values throughout training. This yields an optimization mechanism that modulates the geometry of weight matrices while keeping their spectral norm fixed. We derive the Pion update rule, systematically examine its design choices, and analyze its convergence behavior along with several key properties. Empirical results show that Pion offers a stable and competitive alternative to standard optimizers for both LLM pretraining and finetuning.

Key points

Spectrum preservation: Pion updates weight matrices via left and right orthogonal transformations, keeping singular values (and hence the spectral norm) fixed during the entire training process.
Alternative paradigm: Contrast with additive optimizers (Adam, Muon) — Pion reshapes the geometry of weight matrices rather than adding increments to the weights.
Theoretical grounding: The paper derives the update rule, examines design choices systematically, and provides convergence analysis plus key properties.
Empirical results: Pion is a stable and competitive alternative to standard optimizers for LLM pretraining and finetuning.