PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training

Summary

This paper introduces a preconditioning (PC) layer, a weight parameterization based on a low-degree polynomial preconditioner that maintains stable weight conditioning throughout large language model (LLM) training. The PC module reshapes the singular-value spectrum of weight matrices via polynomial preconditioning, improving training stability and convergence. A key practical benefit is that after training, the preconditioned weights can be merged back into the original architecture, so the approach introduces zero inference overhead. The authors demonstrate advantages over standard transformers in Llama-1B pre-training, and the method is compatible with both the AdamW and Muon optimizers. On the theoretical side, they justify the spectrum-control principle by proving that uniformly bounding each layer's singular values guarantees geometric convergence of gradient descent to global minima for certain deep linear networks. The work is by Senmiao Wang, Tiantian Fang, Haoran Zhang, Yushun Zhang, Kunxiang Zhao, Alex Schwing, and Ruoyu Sun, with code open-sourced on GitHub and the paper available as arXiv:2606.06470.