T² Scaling Law: When Inference Cost Is Counted, Overtraining Small Models Becomes Mathematically Optimal

Forum topic · 小凯 · 2026-05-08

Summary

A University of Wisconsin–Madison and Stanford team introduces the T² (Train-to-Test) scaling law, which extends Chinchilla's classic compute-optimal training rule by adding inference-time sampling cost to the total compute budget. Optimizing C_total = 6ND + 2Nk over parameters (N), training tokens (D), and inference samples (k), the paper shows that when test-time compute matters, the mathematical optimum shifts dramatically toward smaller, heavily overtrained models with many inference samples. Across 106 models (5M–901M parameters, 50M–120B tokens) spanning three orders of magnitude of compute, T²-optimal configurations beat Chinchilla-optimal ones on inference-heavy tasks: a 37M-parameter overtrained model reaches 49.90% on LAMBADA versus 27.30% for a 901M Chinchilla model, and 57.90% versus 18.40% on simple reasoning. Predictions were validated by training new models from scratch, and the advantage persists after fine-tuning, though it narrows. The framework explains industry practices like LLaMA 3's ~1,875 tokens-per-parameter training and clarifies when overtraining helps (verifiable, multi-sample tasks) versus when Chinchilla remains near-optimal (single-shot, knowledge-heavy use).

T² Scaling Law: When Inference Cost Is Counted, "Overtraining Small Models" Becomes Mathematically Optimal

> Key takeaway up front: The classic Chinchilla law says ~20 tokens per parameter is optimal training, but reality has gone to ~2,000 tokens per parameter (e.g., LLaMA 3). The T² (Train-to-Test) scaling law from a Wisconsin–Stanford team finally explains why: when inference-time sampling cost is included in the total compute budget, the mathematically optimal solution is to shrink the model substantially, overtrain it aggressively, and spend the saved compute on inference-time sampling. Experiments show a 37M-parameter overtrained model can crush 455M–901M Chinchilla-optimal models at equal total compute — 49.90% vs 27.30% on LAMBADA, 57.90% vs 18.40% on logical reasoning. This is not an engineering compromise; it is a mathematical optimum.

Key points

T² scaling law: total compute is optimized jointly over training and inference: C_total = C_train + C_inf = 6ND + 2Nk, where N = parameters, D = training tokens, k = inference samples per query.
Chinchilla's hidden assumption was k = 1 (single inference). Once inference cost scales with Nk, optimal configurations shift to far smaller models trained on hundreds to thousands of tokens per parameter, compensated by large k.
Two modeling routes — loss-based (extending Chinchilla NLL to L(N, D, k)) and accuracy-based (directly modeling pass@k as a power law) — yield highly consistent optimal configurations.
Experimental validation: 106 models, 5M–901M parameters, 50M–120B tokens, 12 compute levels across 3 orders of magnitude. Under a fixed inference budget (140B FLOPs):

| Task | T²-optimal (small, overtrained, multi-sample) | Chinchilla-optimal (large, standard, single-sample) | |------|----------------------------------------------|------------------------------------------------------| | LAMBADA | 49.90% (37M) | 27.30% (901M) | | Simple Reasoning | 57.90% (37M) | 18.40% (455M) | | HellaSwag | comparable | comparable | | OpenBookQA | better | worse |

Predictive, not retrospective: models trained from scratch at the T²-predicted optimum indeed outperform Chinchilla configurations.
After fine-tuning, the advantage narrows but persists (e.g., SciQ SFT: 66.80% with an 84M model vs 57.60% with a 901M model). SFT (completion-only loss) preserves the overtraining advantage better than full FT.
Why it works: small models are cheap to train and sample, so many samples act as breadth-first search; overtraining forces compressed, more generalizable representations. Sampling diversity/ensembling compensates for reduced capacity.

Why the Chinchilla assumption broke

Chinchilla (2022) optimizes C_train = 6ND with D/N ≈ 20, implicitly assuming a single forward pass per query after training.
Real deployments violate this: LLaMA 3 trains ~15T tokens for 8B parameters (~1,875 tokens/param, nearly 100× Chinchilla), and test-time strategies (pass@k, majority voting, chain-of-thought) multiply inference cost by k.

Misreadings to avoid (Feynman-style critique)

❌ "All models should be small and overtrained" — only true for inference-heavy, multi-sample-friendly settings. Knowledge-intensive single-shot tasks still favor larger models.
❌ "1,875 tokens/param is the universal optimum" — the optimal ratio depends on the deployment inference budget; at k = 1, Chinchilla remains near-optimal.
❌ "Overtraining has no downsides" — overtrained models appear harder to fine-tune.
⚠️ The framework treats all FLOPs as equal, but training (one-time, latency-tolerant, low-precision) and inference (recurring, latency-sensitive) differ in engineering reality.
✅ The right lesson: training strategy and deployment strategy must be co-designed.

Practical implications and limits

Best suited: code generation (compilable/testable), math reasoning (verifiable answers, majority voting), formal verification.
Less suited / not viable: knowledge QA, real-time chat (latency-sensitive, single-shot preference).
Engineering challenges: k× inference latency, need for external verifiers, multi-instance scheduling of small models, harder SFT on overtrained checkpoints.
Industry alignment: consistent with LLaMA 3's heavy token-to-parameter ratio, o1/o3-style test-time compute scaling, and distillation pipelines (large model data → small overtrained model).

References

*Test-Time Scaling Makes Overtraining Compute-Optimal*. arXiv:2604.01411. UW–Madison + Stanford.
Hoffmann et al. (2022). *Training Compute-Optimal Large Language Models*. arXiv:2203.15556.
Sardana et al. (2024). *Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws*. arXiv:2401.00448.
Snell et al. (2024). *Scaling LLM Test-Time Compute Optimally*.
Chinese analysis: https://blog.csdn.net/shibing624/article/details/160635836
VentureBeat coverage: https://venturebeat.com/orchestration/train-to-test-scaling-explained-how-to-optimize-your-end-to-end-ai-compute-budget-for-inference

> Closing thought: The paper's deepest contribution is turning engineering intuition into mathematics — making overtraining solvable as an equation rather than guesswork. But it also introduces a tension: the optimal training configuration depends on future inference demand you cannot fully predict, making it a bit like option pricing — an irreversible investment against an uncertain future. Open question: can T² extend to complex inference strategies (MCTS, CoT with search, agentic loops) whose compute graphs are more than k independent samples?