[论文] PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Trai

论文概要

研究领域: ML 作者: Senmiao Wang, Tiantian Fang, Haoran Zhang, Yushun Zhang, Kunxiang Zhao, Alex Schwing, Ruoyu Sun 发布时间: 2026-06-04 arXiv: 2606.06470

中文摘要

我们提出了一种预条件（PC）层，通过多项式预条件器对权重进行参数化，确保在大语言模型训练全过程中保持稳定的权重条件。PC 模块通过低次多项式预条件重塑权重矩阵的奇异值谱。训练完成后，预条件化的权重可以合并回原始架构，不引入任何推理开销。我们在 Llama-1B 预训练中展示了所提出的 PC 层相对于标准 Transformer 的优势，同时适用于 AdamW 和 Muon 优化器。理论上，我们通过证明对每一层奇异值进行一致有界可以确保梯度下降对某些深度线性网络的几何收敛到全局最小值，从而证明了这种谱控制原理的合理性。代码已开源：https://github.com/Empath-aln/PC-layer。

原文摘要

We propose a preconditioning (PC) layer, a weight parameterization via polynomial preconditioner that ensures stable weight conditioning throughout LLM training. The PC module reshapes the singular-value spectrum of weight matrices via low-degree polynomial preconditioning. After training, the preconditioned weights can be merged back into the original architecture, incurring no inference overhead. We demonstrate the advantage of the proposed PC layer over standard transformers in Llama-1B pre-training, for both the AdamW and Muon optimizers. Theoretically, we justify this spectrum-control principle by proving that uniformly bounding each layer's singular values ensures geometric convergence of gradient descent to global minima, for certain deep linear networks. Our code is available at ht...

--- *自动采集于 2026-06-08*

#论文 #arXiv #ML #小凯