Learning is Forgetting: LLM Training as Lossy Compression (ICLR 2026)
> Training an LLM is not "memorizing" the internet — it is "forgetting" it, keeping only what is useful for predicting the next token. > > This ICLR 2026 paper uses an information-theoretic microscope to show, for the first time, what actually happens inside a model during training.
1. The Core Question: What Are We Training?
Henry Conklin (Princeton) and the Cohere team ask a deceptively simple question: what happens to a model's representational space during LLM training?
Existing work looks at behavior, local mechanisms (individual attention heads), or static snapshots of finished models — but a dynamic, holistic, theory-driven account of training has been missing. The paper's entry point: treat LLM training as lossy compression.
2. The Lossy-Compression Intuition
- Lossless compression (like ZIP) keeps everything, only re-encoded more efficiently.
- Lossy compression (like MP3 or JPEG) keeps only what matters for a goal — MP3 drops inaudible frequencies, JPEG drops indistinguishable color differences.
- x-axis I(X;Z): how much input information the representation retains (Complexity)
- y-axis I(Y;Z): how much task-relevant information it retains (Expressivity)
- 7B and 32B complete both phases and converge near the boundary.
- 1B fits fine but cannot compress — I(X;Z) stays high, never approaching the boundary.
- Preference information content significantly predicts downstream performance (47 LLMs, r=0.76, p<0.001).
- Assess a model by measuring its Optimality, not running benchmarks
- Quantify "how aligned" a model is via preference information
- Check whether a 7B model is worth continued training by seeing if the compression phase has begun
- Entropy estimation bias: all methods underestimate true entropy; comparisons are relative, not absolute.
- Pretraining only: SFT/RLHF compression dynamics are unexplored — does alignment improve or damage pretrained compression?
- Architecture limits: only decoder-only Transformers tested; MoE, State Space Models, and RWKV remain unverified.
- Causality: correlation is not causation — does compression cause generalization, or do well-generalizing models merely compress better? Intervention experiments needed.
arxiv.org/abs/2604.07569— paper (ICLR 2026)github.com/hcoxec/soft_h— code (soft-entropy estimator)- Tishby & Zaslavsky (2015) — foundational Information Bottleneck theory
- Shwartz-Ziv & Tishby (2017) — IB empirics on MNIST
- Feldman (2016) — cognitive science review of compression as learning
- OLMo2 model family — Allen AI open checkpoints
LLM training reads far more text than a human could in 200 lifetimes and compresses it into billions of parameters. Details useless for next-token prediction are progressively forgotten. Hence the title: Learning is Forgetting.
3. Information Bottleneck: Two Phases of Training
The theoretical foundation is Information Bottleneck (IB) theory (Tishby & Zaslavsky, 2015). Plot:
The optimal compression boundary is the diagonal I(X;Z) = I(Y;Z) — every retained bit of input yields a bit of expressivity, with no waste. IB theory predicts two training phases:
Phase 1: Fitting
The model rapidly absorbs information; I(Y;Z) rises quickly as it learns predictive representations, while I(X;Z) also rises.Phase 2: Compression
After the training loss saturates, I(X;Z) begins to fall while I(Y;Z) stays stable or rises slowly. The representation space is denoising, approaching the optimal boundary.Key insight: the compression phase is where generalization emerges — like a student rewriting notes in their own words rather than memorizing them.
4. OLMo2 7B: A Clean Experimental Validation
Using OLMo2 7B (open model with intermediate checkpoints), the training trajectory on the information plane matches IB predictions: first moving up-right (fitting), then bending up-left (compression), converging toward the diagonal. When the model nears the boundary, next-token loss saturates — compression is not a side effect but a natural outcome of the training objective.
The pattern also holds for SmolLM2 and Pythia: different architectures, different scales, same behavior.
5. Scale Effects: Small Models Get Stuck
Comparing 1B, 7B, and 32B models:
Small models suffer a "capacity crunch": no spare room to organize information. 7B appears to be a threshold — beyond it, a model has capacity not only to learn but also to *forget*. Forgetting is an advanced capability.
6. Compression Optimality Predicts Performance
The most practical finding. The authors define:
Optimality = Expressivity / Complexity = I(Y;Z) / I(X;Z)
Closer to 1.0 means more optimal compression. Validated on 6 benchmarks × 6 model families (Qwen2.5, Gemma2, Mistral, OLMo2, Pythia, SmolLM2) spanning reasoning, code, math, and language understanding, Optimality correlates significantly with downstream performance — you can predict a model's usefulness from its position on the information plane without running benchmarks.
7. Preference Information: A Code for Alignment
Using the Tulu preference dataset (each prompt with a preferred and a rejected response), they compute I(Z; preferred):
Stronger human-preference signal in representations means better real-world performance — an information-theoretic justification for RLHF/DPO: alignment is not decorative post-processing but an intrinsic property of representational structure. The paper calls this quantifying "aligned representations."
8. Methodological Innovation: The Soft-Entropy Estimator
Measuring mutual information in LLMs is notoriously hard — binning fails at 4096–16384 dimensions. The paper uses a soft-entropy estimator (Conklin 2025):
1. Normalize representation vectors to the unit sphere 2. Sample n random anchor points on the sphere 3. Use softmax to get a soft assignment of each representation to anchors 4. Take the Shannon entropy of these distributions
A "soft quantization" — say "30% in bin A, 70% in bin B" instead of hard binning. Differentiable, scalable to any dimension. This is the first operationalization of IB theory at LLM scale.
9. Why This Paper Matters
1. Theoretical Unification
Connects deep learning, information theory, and cognitive science. Compression as learning — human infants also compress sensory input when acquiring language (Feldman 2016). A formal bridge between LLMs and human cognition.2. Actionable Insights
3. Training Guidance
If small models (<7B) never enter the compression phase, options are: more parameters, more tokens, or a modified objective that incentivizes "organizing."10. Limitations and Open Questions
11. One-Sentence Summary
> LLM training is not a memorization contest but an art of compression. Models learn by forgetting — keeping only prediction-useful information. And the metric of how gracefully they forget (Optimality) directly predicts real-world performance. > > The paper cracks the black box open a little: we finally know what those billions of parameters actually contain.