Requential Coding: Pushing the Limits of Model Compression with Student-Selected Samples

Forum topic · 小凯 · 2026-07-15

Summary

Requential coding is a new model compression and generalization analysis method introduced by Shikai Qiu, Marc Finzi, Yujia Zheng, Kun Zhang, and Andrew Gordon Wilson (arXiv:2607.11883). Unlike parameter-based approaches such as quantization, whose code lengths scale with model size, and prequential coding, which codes the exact data sequence regardless of what the model learns, requential coding uses a teacher model to select training samples from the student's own distribution. The student's code records only these choices and spends bits solely where teacher and student disagree. As a result, code length is independent of parameter count and data entropy, often orders of magnitude shorter than prequential codes, with advantages growing at scale. Plugging this code into PAC-Bayes bounds yields state-of-the-art generalization guarantees for billion-parameter LLMs, outperforming bounds from aggressive quantization. The same code predicts gradual overfitting over multiple training epochs and separates learnable information from unpredictable randomness, revealing that low-entropy text contains far more learnable structure than high-entropy image data.

Paper Overview

Field: Machine Learning Authors: Shikai Qiu, Marc Finzi, Yujia Zheng, Kun Zhang, Andrew Gordon Wilson Published: 2026-07-13 arXiv: 2607.11883

Introduction

Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization. Large neural networks may learn functions far simpler than their parameter counts suggest, but it is challenging to construct codes that realize this simplicity.

Limitations of Existing Approaches

Parameter-based methods (e.g., quantization): code lengths scale with model size and are insensitive to how much information the parameters actually store.
Prequential coding: compresses the training trajectory to bypass model-size scaling, but codes the exact data sequence regardless of how much the model learns, yielding large codes when the data has high entropy.

Requential Coding

The authors introduce requential coding: a teacher model selects training samples drawn from the student's own distribution, and the student's code records only these selections, spending bits only where the teacher and student disagree.

Key properties:

Code length is independent of parameter count and data entropy.
Codes are often orders of magnitude shorter than prequential counterparts, with the advantage growing with scale.

Generalization Guarantees

Substituting requential codes into PAC-Bayes bounds provides state-of-the-art generalization guarantees for billion-parameter LLMs. Even assuming zero training error, the resulting bounds improve on bounds built from aggressively quantized models. In the compute-optimal regime, the bound tightens as models scale, because models become increasingly compressible relative to dataset size.

Additional Findings

The same code predicts gradual overfitting across multiple training epochs.
It separates learnable information from unpredictable, random content in a dataset, revealing that low-entropy text contains far more learnable structure than high-entropy image data.

---

*Auto-collected on 2026-07-15. Original abstract: https://arxiv.org/abs/2607.11883*