Requential Coding: Pushing the Limits of Model Compression with Self-Selected Training Samples

Forum topic · 小凯 · 2026-07-15

Summary

Requential coding is a new model-compression framework introduced by Shikai Qiu, Marc Finzi, Yujia Zheng, Kun Zhang, and Andrew Gordon Wilson (arXiv:2607.11883). Unlike parameter-based methods such as quantization—whose code lengths scale with model size—and prequential coding—which encodes the exact data sequence regardless of model learning—the requential approach has a teacher model select training samples drawn from the student's own distribution. The student's code records only these selections, spending bits only where teacher and student disagree. As a result, code length is independent of both parameter count and data entropy, often orders of magnitude shorter than prequential equivalents, with advantages growing with scale. Plugging the code into PAC-Bayes bounds yields state-of-the-art generalization guarantees for billion-parameter LLMs, beating bounds built even on aggressively quantized models, and the bound tightens with scale in the compute-optimal regime. The same code predicts gradual overfitting over multiple training epochs and separates learnable information from unpredictable stochastic content, revealing that low-entropy text contains far more learnable structure than high-entropy image data.

Requential Coding: Pushing the Limits of Model Compression

Field: Machine Learning Authors: Shikai Qiu, Marc Finzi, Yujia Zheng, Kun Zhang, Andrew Gordon Wilson Published: 2026-07-13 arXiv: 2607.11883

Abstract (translated from the forum post)

Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization. Large neural networks may learn functions far simpler than their parameter counts suggest, but constructing codes that realize this simplicity is challenging. Parameter-based methods such as quantization produce code lengths that scale with model size, insensitive to how much information the parameters actually store. Prequential coding bypasses this issue by compressing the training trajectory, but it encodes the exact data sequence regardless of how much the model learns, yielding large codes when the data has high entropy.

The paper introduces requential coding: a teacher model selects training samples drawn from the student's own distribution, and the student's code records only these selections, spending bits only where teacher and student disagree. Consequently, code length is independent of both parameter count and data entropy, often orders of magnitude shorter than prequential counterparts, with the advantage growing with scale.

Substituting this code into PAC-Bayes bounds provides state-of-the-art generalization guarantees for billion-parameter LLMs, outperforming bounds built even under an assumption of zero error on aggressively quantized models. In the compute-optimal regime, the bound tightens with scale, as models become increasingly compressible relative to dataset size. The same code also predicts gradual overfitting over multiple training epochs and separates learnable information from unpredictable stochastic content in a dataset, revealing that low-entropy text contains far more learnable structure than high-entropy image data.

Key takeaways

Requential coding: teacher selects samples from the student's own distribution; the code only encodes disagreements, not the raw data sequence.
Scale-independent codes: code length depends on neither parameter count nor data entropy.
Sharper PAC-Bayes bounds: best known generalization guarantees for billion-parameter LLMs; tighter at scale in the compute-optimal regime.
Additional uses: predicts overfitting across training epochs; disentangles learnable structure from irreducible noise, comparing text vs. image datasets.

*Auto-collected on 2026-07-15.*