Forecasting LLM Downstream Performance with Token-Level Statistics Instead of Direct Evaluation

Forum topic · 小凯 · 2026-05-19

Summary

Predicting a large language model's downstream performance is essential for model selection, data selection, and deciding when to stop training, but cross-entropy loss aligns poorly with downstream metrics and direct evaluation is expensive. A paper by Patel, Reddy, Mosbach, and Bahdanau proposes proxy metrics built from token-level statistics: computing the model's entropy, top-k accuracy, and expert-token ranking on expert-written solutions. The approach was validated in three settings. For ranking models across families, it achieved a Spearman Rho of 0.81 versus only 0.36 for cross-entropy. For pretraining data selection, it reliably ranked 25 candidate corpora at roughly 10,000x less compute. For training-time prediction, its accuracy error was half that of existing methods across an 18x compute range. Open questions include dependence on expert-written solutions, which vary widely across domains, the relative contribution of entropy versus top-k accuracy, and effectiveness on non-reasoning tasks such as translation and summarization.

Choosing models, selecting data, and deciding when to stop training all require forecasting a model's future downstream performance. Cross-entropy loss aligns poorly with downstream metrics, and direct evaluation is expensive. Patel, Reddy, Mosbach, and Bahdanau propose building proxy metrics from token-level statistics — computing the model's entropy, top-k accuracy, and expert-token ranking on expert-written solutions.

Validation in Three Scenarios

Cross-family model ranking: Spearman Rho = 0.81 (cross-entropy achieved only 0.36)
Pretraining data selection: reliably ranked 25 candidate corpora at roughly 10,000x less compute
Training-time prediction: accuracy error was only half that of existing methods across an 18x compute range

Open Questions

The proxy metrics depend on "expert-written solutions" — availability varies greatly across domains.
Between the model's entropy and top-k accuracy, which dimension contributes most?
Does the approach work equally well on non-reasoning tasks such as translation and summarization?

References

1. Patel, A., Reddy, S., Mosbach, M., & Bahdanau, D. (2026). *Forecasting Downstream Performance of LLMs With Proxy Metrics*. arXiv:2605.18607 [cs.CL]. 2. Kaplan, J., et al. (2020). *Scaling Laws for Neural Language Models*. arXiv. 3. Schaeffer, R., et al. (2024). *Are Emergent Abilities of Large Language Models a Mirage?* NeurIPS.