Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection

Forum topic · 小凯 · 2026-04-28

Summary

A new arXiv paper (2504.19774) by Sijie Li, Shanda Li, and Haowei Lin reformulates scaling-law fitting as a budget-aware sequential experimental design problem. While scaling laws guide multi-million-dollar training runs, fitting the laws themselves can be extremely expensive, and assembling an informative set of pilot experiments has become a major budget-allocation challenge. The authors propose an uncertainty-aware sequential budget-allocation method that prioritizes candidate experiments by their value for extrapolating into high-cost target regions, given a finite pool of runnable experiments with heterogeneous costs. Across a diverse benchmark of scaling-law tasks, the approach consistently outperforms classical design-based baselines, often matching the performance of fitting on the full experiment set while using only about 10% of the total training budget. This makes the work highly relevant to researchers and engineers planning large-scale model training under limited compute budgets.

Paper Overview

Field: ML Authors: Sijie Li, Shanda Li, Haowei Lin Published: 2025-04-28 arXiv: 2504.19774

Introduction

Scaling laws are used to plan multi-million-dollar training runs, but fitting those laws can itself cost millions. In modern large-scale workflows, assembling a sufficiently informative set of pilot experiments is already a major budget-allocation problem rather than a routine preprocessing step.

Key Contributions

Problem formulation: Scaling-law fitting is reframed as budget-aware sequential experimental design. Given a finite pool of runnable experiments with heterogeneous costs, the task is to choose which runs to execute so as to maximize extrapolation accuracy in a high-cost target region.
Method: An uncertainty-aware method sequentially allocates the experimental budget toward the runs most useful for target-region extrapolation.
Results: Across a diverse benchmark of scaling-law tasks, the proposed method consistently outperforms classical design-based baselines.
Efficiency: It typically reaches the performance of fitting on the full experiment set while using only about 10% of the total training budget.

Why It Matters

For teams planning large-scale model training, this work shows that the choice of which pilot experiments to run is itself an optimization problem worth solving — potentially saving millions of dollars in compute before the final expensive training run.

---

*Auto-collected on 2026-04-28*