Requential Coding: How AI Compresses Knowledge by Training on Its Own Predictions

*English translation/structured summary of a Chinese forum post interpreting the paper "Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data" (Qiu et al., CMU & NYU).*

Key points

• Compression is fundamental to intelligence. Any system that compresses data well must have discovered regularities in it — a core insight from Solomonoff (1964), Kolmogorov (1965), and MDL (Rissanen, 1978). The shortest description equals the best understanding.
• Traditional methods compress parameters, not knowledge. Quantization keeps code length proportional to parameter count; pruning discards weight structure; knowledge distillation mimics behavior. A trillion-parameter model may implement a function representable with far fewer parameters, but naive shrinking destroys performance.
• Prequential coding compresses the *training trajectory*: at each step the model predicts the next data point and only the residual error is encoded. But it still encodes the entire data sequence — if the data is high-entropy, the code stays long even when the model has learned a lot.
• Requential Coding's core idea: let the teacher sample training examples from the student's own predicted distribution, selecting cases where teacher and student disagree most. Only these disagreement points need to be encoded; agreement points cost nothing. Code length therefore reflects what the model *learned*, not dataset size.
• Headline result: the resulting code length is independent of parameter count and data entropy, and often orders of magnitude shorter than the prequential counterpart.
• Counterintuitive scaling: at fixed training loss, larger models and even model ensembles compress to *smaller* code sizes despite having more parameters — bigger models capture deeper, more compressible structure.
• PAC-Bayes generalization guarantees: using requential codes as the complexity term yields state-of-the-art bounds for billion-parameter LLMs — tighter even than bounds assuming error-free aggressive quantization, and the bound tightens as model scale grows in the compute-optimal regime.
• Overfitting detection: code length increases when a model begins memorizing noise across multiple training epochs, providing a natural overfitting signal.
• Text vs. images: lower-entropy text holds far more *learnable* structure than higher-entropy image data, whose apparent complexity is largely unlearnable noise.

Why it matters

The post argues Requential Coding builds a provable bridge between compression ↔ generalization ↔ understanding. It suggests a new model-selection criterion — compressibility at fixed performance — and future directions such as adaptive data selection during training, information-theoretic model merging, and using compression ratios to quantify the signal-to-noise content of scientific data.

As the author puts it, extending Feynman: *"What I cannot compress, I do not truly understand."*

References cited in the post

• Qiu, S., et al. "Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data." arXiv:2607.11883, 2026.
• Solomonoff, R. J. "A Formal Theory of Inductive Inference." *Information and Control*, 1964.
• Kolmogorov, A. N. "Three Approaches to the Quantitative Definition of Information." *Problems of Information Transmission*, 1965.
• Rissanen, J. "Modeling by Shortest Data Description." *Automatica*, 1978.
• Blier, L., & Ollivier, Y. "The Description Length of Deep Learning Models." NeurIPS, 2018.