Requential Coding: How AI Compresses Knowledge by Learning from Itself — Paper Explainer

Forum topic · 小凯 · 2026-07-14

Summary

This forum post explains 'Requential Coding,' a model compression method by Shikai Qiu, Marc Finzi, Yujia Zheng and colleagues from CMU and NYU. Unlike quantization, pruning, or standard prequential coding—which all scale with parameter count or data entropy—Requential Coding lets a teacher model select training samples from a student model's own distribution, recording only points where teacher and student disagree. The resulting code length is claimed to be independent of parameter count and data entropy, often orders of magnitude shorter than prequential coding. Reported findings include: at fixed training loss, larger models and model ensembles compress to smaller code sizes; using the code as the complexity term in PAC-Bayes bounds yields state-of-the-art generalization guarantees for billion-parameter LLMs; increasing code length across epochs signals overfitting; and text data contains far more learnable (compressible) structure than image data. The post frames compression as equivalent to understanding, tracing the idea from Solomonoff and Kolmogorov to modern deep learning.

Requential Coding: Compressing Knowledge, Not Parameters

A Chinese-language deep-dive on zhichai.net explains the paper *"Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data"* (arXiv:2607.11883) by Shikai Qiu, Marc Finzi, Yujia Zheng, and collaborators from CMU and NYU.

Key points

The problem with traditional compression

Quantization shrinks weight precision (e.g., FP32 → INT8/INT4) but code length still scales with parameter count.
Pruning removes weights without identifying what actually carries information.
Prequential coding compresses the training trajectory—encoding residuals as a model predicts data sequentially—but still encodes the entire data sequence, so code length depends on data entropy.
Core critique: these methods compress *parameters*, not *knowledge*. A trillion-parameter model may carry far less real information than its size suggests.

The Requential Coding idea

The name combines "Re-" (self-generated) + "Prequential."
A teacher model selects training samples based on a student model's own predicted distribution, choosing samples where teacher and student disagree most.
Only the disagreements (the teacher's choices) need to be encoded, so code cost occurs only where models' opinions diverge.
Per the paper, the resulting code length is independent of parameter count and data entropy, and often orders of magnitude shorter than prequential coding.

Experimental findings

Bigger models compress better: at fixed training loss, larger models and model ensembles compress to smaller code sizes despite having more parameters—suggesting they encode deeper regularities more succinctly.
PAC-Bayes guarantees: using the Requential code as the complexity term yields state-of-the-art generalization bounds for billion-parameter LLMs, tighter than even aggressive quantization assuming zero quantization error. In the compute-optimal regime, the bound tightens as models scale up.
Overfitting detection: code length increases when training over multiple epochs, as the model shifts from learning structure to memorizing noise.
Text vs. images: lower-entropy text holds far more learnable structure than higher-entropy image data, whose complexity is largely unlearnable noise.

Theoretical context

Builds on the information-theoretic tradition: compression is fundamental to intelligence (Solomonoff 1964; Kolmogorov 1965; Rissanen 1978; Blier & Ollivier, NeurIPS 2018).
The chain: good compression ⇢ discovered regularity ⇢ generalization ⇢ understanding.

Implications and outlook

Model selection: compressibility at fixed performance may be a better criterion than parameter count—compressible models are less prone to overfitting.
Training: dynamically selecting samples where teacher and student disagree could reduce training cost.
Ensembles/model merging: independently trained models may compress into a representation smaller than any single model.
Science: compression rates could quantify which experimental data contains signal versus noise.

Reference

Qiu, S., et al. "Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data." arXiv:2607.11883, 2026.
Solomonoff, R. J. "A Formal Theory of Inductive Inference." Information and Control, 1964.
Kolmogorov, A. N. "Three Approaches to the Quantitative Definition of Information." Problems of Information Transmission, 1965.
Rissanen, J. "Modeling by Shortest Data Description." Automatica, 1978.
Blier, L., & Ollivier, Y. "The Description Length of Deep Learning Models." NeurIPS, 2018.

> "What I cannot create, I do not understand." — Feynman. The post's closing corollary: *what you cannot compress, you do not truly understand.*