> Training an LLM is not "memorizing" the internet — it is "forgetting" — keeping only information useful for predicting the next token. > > This ICLR 2026 paper uses an information-theoretic microscope to show, for the first time, what actually happens inside a model during training.
The Core Question: What Are We Training?
Henry Conklin (Princeton) and the Cohere team ask a deceptively simple question: what happens to a model's representational space during LLM training?
Existing work looks at behavior, local components (individual attention heads), or static snapshots of finished models. A dynamic, holistic, theory-driven description of training has been missing.
The paper's clever entry point: treat LLM training as lossy compression.
The Lossy Compression Intuition
- Lossless compression (like ZIP) reorganizes information without losing anything.
- Lossy compression (like MP3 or JPEG) keeps only what matters for the goal.
- X-axis I(X;Z): complexity — how much input information the representation retains
- Y-axis I(Y;Z): expressivity — how much information useful for prediction it retains
- 7B and 32B complete both phases and converge near the boundary.
- 1B fits fine but cannot compress — I(X;Z) stays high, far from the boundary.
- Judge a model by its Optimality, without running downstream benchmarks
- Quantify alignment via preference information
- Check whether a sub-7B model ever enters the compression phase
- Entropy estimators are biased: all methods underestimate true entropy; comparisons are relative, not absolute.
- Pretraining only: SFT/RLHF compression dynamics are unexplored.
- Architecture limits: only decoder-only Transformers tested; MoE, State Space Models, RWKV remain open.
- Causality: correlation ≠ causation — intervention experiments are needed.
arxiv.org/abs/2604.07569— paper (ICLR 2026)github.com/hcoxec/soft_h— code (soft-entropy estimator)- Tishby & Zaslavsky (2015) — foundational IB theory
- Shwartz-Ziv & Tishby (2017) — empirical IB on MNIST
- Feldman (2016) — compression-as-learning in cognitive science
- OLMo2 model family — Allen AI open checkpoints
LLM training reads far more text than a human could in 200 lifetimes and compresses it into billions of parameters — but details useless for next-token prediction are gradually forgotten. Hence the title: Learning is Forgetting.
Information Bottleneck: Two Phases of Training
The theoretical foundation is Information Bottleneck (IB) theory (Tishby & Zaslavsky, 2015). On the information plane:
The optimal compression boundary is the diagonal I(X;Z) = I(Y;Z).
Phase 1: Fitting
The model rapidly absorbs information; I(Y;Z) rises quickly as it learns predictive representations, with I(X;Z) rising too.Phase 2: Compression
After training loss saturates, I(X;Z) begins to fall while I(Y;Z) holds steady. The representation space is denoising, moving toward the optimal boundary.Key insight: the compression phase is where generalization emerges — like a student rewriting notes in their own words rather than memorizing them.
Experimental Validation on OLMo2 7B
Using OLMo2 7B (open model with intermediate checkpoints), the training trajectory on the information plane perfectly matches IB predictions: first rushing up-and-right (fitting), then curving up-and-left (compression), approaching the diagonal. As the model nears the boundary, next-token loss saturates — compression is not a side effect but a natural consequence of the training objective.
The pattern also holds for SmolLM2 and Pythia across architectures and scales.
Scale Effects: The Small-Model Bottleneck
Comparing 1B, 7B, and 32B models:
Small models have "capacity anxiety": no room for information housekeeping. Around 7B is a threshold above which models have capacity not only to learn but to *forget*.
Optimality = Performance Predictor
The paper's most practical finding. Define:
Optimality = Expressivity / Complexity = I(Y;Z) / I(X;Z)
Values near 1.0 mean near-optimal compression. Validated across 6 benchmarks × 6 model families (Qwen2.5, Gemma2, Mistral, OLMo2, Pythia, SmolLM2) covering reasoning, code, math, and language understanding, Optimality correlates significantly with downstream performance. You can predict how good a model is just from its position on the information plane.
Preference Information: The Code of Alignment
Using the Tulu preference dataset (each prompt has a preferred and a rejected response), the paper measures I(Z; preferred).
Finding: preference information significantly predicts downstream performance (47 LLMs, r=0.76, p<0.001). From an information-theoretic view, this supports RLHF/DPO — alignment is an intrinsic property of representational structure, not post-hoc decoration. The paper calls this quantifying "aligned representations."
Methodological Innovation: The Soft-Entropy Estimator
Measuring mutual information in LLMs is a nightmare: binning representations fails at 4096–16384 dimensions. The soft-entropy estimator (Conklin 2025):
1. Normalize representation vectors to the unit sphere 2. Sample n random anchor points on the sphere 3. Use softmax to compute soft assignments of each representation to anchors 4. Take the Shannon entropy of these distributions
A "soft quantization" — differentiable, scalable to any dimension. This is the first operationalization of IB theory at LLM scale.
Why This Paper Matters
1. Theoretical unification
Connects deep learning, information theory, and cognitive science. Compression as learning echoes how human infants compress sensory input (Feldman 2016) — a formal bridge between LLMs and human cognition.2. Actionable insights
3. Guidance for training
If small models never compress, the fix is more parameters, more tokens, or a modified objective.Limitations and Open Questions
One-Sentence Summary
> LLM training is not a memorization contest but the art of compression. Models learn by forgetting — keeping only prediction-relevant information — and the Optimality metric measuring how elegantly they forget directly predicts real-world performance.