[论文] Generalization behavior of OPTQ and the role of regularization

论文概要

研究领域: ML
作者: Erin George, Rayan Saab
发布时间: 2026-09-25
arXiv: 2609.31560

中文摘要

大型神经网络可以通过将其权重舍入或“量化”为可用更少比特表示的数来压缩。OPTQ 是一种量化算法，它逐步量化神经网络权重，使指定校准数据集上的平方量化误差尽可能小。我们在泛化设定下研究 OPTQ 及其变体算法随机 OPTQ 的性能，并推导当测试点从固定分布抽取时算法累积期望平方误差的界。我们证明两个结果：其一，将泛化误差与由同分布独立样本构成的校准数据集上的误差联系起来；其二，对所有“足够好”的分布，无论校准数据集如何，都界定随机 OPTQ 的泛化误差。在这两个结果中，正则化项 λ 都扮演重要角色。我们利用这些结果的洞见对 λ 的选择提出新建议，实验表明该选择在性能上优于文献中的既有建议。

原文摘要

Large neural networks can be compressed by rounding or "quantizing" their weights to numbers that admit representations with fewer bits. One algorithm for quantization, OPTQ, progressively quantizes the weights of a neural network so that the squared quantization error on a specified calibration dataset is as small as possible. We study the performance of OPTQ and a variant algorithm, stochastic OPTQ, in a generalization setting and derive bounds for the expected squared error accrued by the algorithm when a test point is drawn from a fixed distribution. We prove two results. One result relates the generalization error to the error on a calibration dataset comprising independent samples from the same distribution as the test distribution. The other result bounds the generalization error of...

自动采集于 2026-09-29

#论文 #arXiv #ML #小凯