[论文] Requential Coding: Pushing the Limits of Model Compression with S...

论文概要

研究领域: ML 作者: Shikai Qiu, Marc Finzi, Yujia Zheng, Kun Zhang, Andrew Gordon Wilson 发布时间: 2026-07-13 arXiv: 2607.11883

中文摘要

压缩是智能的基础。能将训练数据表示为短码的模型，发现了促成泛化的规律。大型神经网络可能学到远比参数量简单的函数，但构造实现这种简洁性的编码颇具挑战。基于参数的方法如量化产生的编码长度随模型规模缩放，对其存储多少信息不敏感。Prequential编码通过压缩训练轨迹绕过此问题，但无论模型学到多少都编码精确数据序列，在数据高熵时产生大编码。本文提出 requential 编码：教师模型从学生自身分布中选择训练样本，学生的编码仅记录这些选择，只在教师与学生意见不一致处消耗比特。编码长度独立于参数量和数据熵，常比prequential对应物短数个数量级，优势随规模增长。将其代入PAC-Bayes界，为十亿参数LLM提供最优泛化保证，即使假设零误差也优于激进量化后构建的界限。在计算最优区域，该界限随规模收紧，因为模型相对数据集大小越来越可压缩。同一编码还预测模型在多轮训练时逐渐过拟合，并分离数据集中可学习信息与不可预测的随机内容，揭示低熵文本比高熵图像数据包含远多可学习结构。

原文摘要

Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization. Large neural networks may learn functions far simpler than their parameter counts suggest, but it is challenging to construct codes that realize this simplicity. Parameter-based methods such as quantization produce code lengths that scale with model size, insensitive to how much information the parameters store. Prequential coding bypasses this issue by compressing the training trajectory, but codes the exact data sequence regardless of how much the model learns, yielding large codes when the data has high entropy. We introduce requential coding, where a teacher model selects training samples drawn from the student's own distributi...

--- *自动采集于 2026-07-15*

#论文 #arXiv #ML #小凯