Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Uncertainty-Aware Sequential Experimental Design

Forum topic · 小凯 · 2026-04-28

Summary

A paper by Sijie Li, Shanda Li, and Haowei Lin (arXiv:2504.19774, April 2025) reformulates scaling-law fitting as a budget-aware sequential experimental design problem. Scaling laws guide multi-million-dollar LLM training runs, yet fitting the laws themselves can be extremely expensive because assembling informative pilot experiments becomes a major budget-allocation task. The authors propose an uncertainty-aware method that sequentially allocates a limited experimental budget toward candidate runs that are most valuable for extrapolating model performance in a high-cost target region. Given a finite pool of runnable experiments with heterogeneous costs, the method chooses which experiments to execute to maximize extrapolation accuracy where it matters most. Across a diverse benchmark of scaling-law tasks, the approach consistently outperforms classical design-based baselines, often matching full-experiment fitting performance while using only about 10% of the total training budget, offering a practical recipe for cost-efficient scaling-law research.

Paper Overview

Field: Machine Learning
Authors: Sijie Li, Shanda Li, Haowei Lin
Published: 2025-04-28
arXiv: 2504.19774

Abstract

Scaling laws are used to plan multi-million-dollar training runs, but fitting those laws can itself cost millions. In modern large-scale workflows, assembling a sufficiently informative set of pilot experiments is already a major budget-allocation problem rather than a routine preprocessing step.

The paper formulates scaling-law fitting as budget-aware sequential experimental design: given a finite pool of runnable experiments with heterogeneous costs, choose which runs to execute so as to maximize extrapolation accuracy in a high-cost target region.

Method

The authors propose an uncertainty-aware method for sequentially allocating the experimental budget toward the runs most useful for target-region extrapolation. Rather than treating pilot experiments as a fixed preprocessing step, the approach adaptively decides which next experiment to run based on its value for improving inference in the costly target region.

Results

Across a diverse benchmark of scaling-law tasks, the method consistently outperforms classical design-based baselines.
It often achieves performance comparable to fitting with the full experiment set while using only ~10% of the total training budget.

Takeaway

Treating scaling-law fitting itself as an optimization problem—selecting experiments under a budget constraint—can dramatically reduce the cost of planning large-scale model training runs.