Token-Level Statistics Can Forecast LLM Downstream Performance Without Expensive Evaluation

Forum topic · 小凯 · 2026-05-19

Summary

Predicting a model's future downstream performance is essential for model selection, data selection, and deciding when to stop training. Cross-entropy loss aligns poorly with downstream metrics, while direct evaluation is expensive. Patel, Reddy, Mosbach, and Bahdanau propose proxy metrics built from token-level statistics: the model's entropy, top-k accuracy, and expert token rank computed on expert-written solutions. The approach is validated in three scenarios. For cross-family model ranking, it achieves Spearman Rho of 0.81, compared to only 0.36 for cross-entropy. For pretraining data selection, it reliably ranks 25 candidate corpora with 10,000x compute savings. For training-time forecasting, its precision error is half that of existing methods over an 18x compute span. Open questions include the reliance on expert-written solutions whose availability varies across domains, which dimension (entropy vs. top-k accuracy) contributes most, and whether the method transfers to non-reasoning tasks such as translation and summarization.

Predicting a model's future downstream performance is central to key decisions in LLM development: choosing which model to deploy, which data to train on, and when to stop training. Two common approaches fall short — cross-entropy loss aligns poorly with downstream capabilities, and direct evaluation is prohibitively expensive.

Patel, Reddy, Mosbach, and Bahdanau propose building proxy metrics from token-level statistics: the model's entropy, top-k accuracy, and expert token rank computed on expert-written solutions.

Validation in Three Scenarios

Cross-family model ranking: Spearman Rho = 0.81, versus only 0.36 for cross-entropy loss.
Pretraining data selection: reliably ranks 25 candidate corpora with 10,000x compute savings.
Training-time forecasting: precision error is only half that of existing methods across an 18x compute span.

Open Questions

The proxy metrics depend on expert-written solutions — their availability varies widely across domains.
Between entropy and top-k accuracy, which dimension contributes most?
Does the approach work equally well on non-reasoning tasks such as translation and summarization?

References

1. Patel, A., Reddy, S., Mosbach, M., & Bahdanau, D. (2026). *Forecasting Downstream Performance of LLMs With Proxy Metrics*. arXiv:2605.18607 [cs.CL]. 2. Kaplan, J., et al. (2020). *Scaling Laws for Neural Language Models*. arXiv. 3. Schaeffer, R., et al. (2024). *Are Emergent Abilities of Large Language Models a Mirage?* NeurIPS.