Paper Overview
Field: Machine Learning Authors: Xin He, Huan Lao, Xiliang Lu, Shijun Zhang Released: 2026-10-22 arXiv: 2610.31663
Summary
Constructing accurate and computationally tractable approximation spaces is a fundamental task in scientific computing. Partially trainable neural networks bridge fixed random features and fully adaptive representations. This paper quantifies a phase transition pattern in approximation bounds and observes analogous behavior in training errors: a rapid but smooth initial improvement followed by diminishing returns, with a sharper transition under greater initialization mismatch.
Under polynomial spectral decay and leverage compatibility conditions, the authors establish a squared L₂ approximation rate for two-layer networks of width m and trainable proportion P:
O((κ_P log(m/δ)/m)^α)
where κ_P quantifies the sampling mismatch between the fixed and trainable groups relative to the target RKHS. The assumptions are verified for spherical biased-ReLU features.
Key Findings
- A phase transition pattern exists in approximation bounds and mirrors training error behavior.
- Numerically, a trainable proportion of only about 4.1% captures 90% of the approximation-bound improvement from P=0 to P=1.
- Networks as deep as 15 layers exhibit similar phase transition behavior.
Abstract (original)
Constructing accurate and computationally tractable approximation spaces is a fundamental task in scientific computing. Partially trainable neural networks bridge fixed random features and fully adaptive representations. This paper quantifies a phase transition pattern in approximation bounds and observes analogous behavior in training errors: a rapid but smooth initial improvement followed by diminishing returns, with a sharper transition under greater initialization mismatch. Under polynomial spectral decay, we establish a squared L₂ approximation rate for two-layer networks and verify these assumptions with spherical biased-ReLU features.
--- *Auto-collected on 2026-10-24*