Trust the Direction, Search the Step: Zero-and-First-Order Methods for Neural Network Optimization

Forum topic · 小凯 · 2026-10-04

Summary

This post summarizes an arXiv paper (2610.02190) by Cristian McGee, El Houcine Bergou, and Aritra Dutta on step-size selection in large-scale neural network optimization. The authors propose ZFO (Zero-and-First-Order optimization), a lightweight framework that decouples direction selection from step-size choice. A trusted first-order optimizer determines the search direction, while zeroth-order evaluations are performed only along this one-dimensional subspace to decide how far to move. Using the current gradient plus two additional objective function evaluations, ZFO builds a local model of the objective along the proposed direction and selects a curvature-aware step within a bounded search interval, at lower cost than a full line search. Theoretical results show that shared sample evaluations yield reliable finite-difference curvature estimates, the induced local model selects near-optimal steps, and ZFO converges to a neighborhood of stationary points. Empirically, ZFO frequently improves optimization and final performance over fixed-step first-order baselines across evaluated language models and datasets. Code is available on GitHub.

Paper Overview

Field: Machine Learning Authors: Cristian McGee, El Houcine Bergou, Aritra Dutta arXiv: 2610.02190

Abstract

Step-size selection remains a central challenge in large-scale neural network optimization: conservative steps slow convergence, while aggressive steps can destabilize it. The authors combine Zero-and-First-Order optimization (ZFO) into a lightweight framework that decouples direction selection from step-size selection.

Key Ideas

Direction from first order: ZFO uses a trusted first-order optimizer to determine the search direction.
Step-size from zero order: Zeroth-order evaluations are performed only along this one-dimensional subspace to choose how far to move.
Low overhead: Using the current gradient information and just two additional objective function evaluations, ZFO constructs a local model of the objective along the proposed direction and selects a curvature-aware step within a bounded search interval — cheaper than a full line search.

Theoretical Guarantees

Shared sample evaluations produce reliable finite-difference curvature estimates.
The induced local model selects a near-optimal step within the search interval.
ZFO converges to a neighborhood of stationary points.

Empirical Results

Across all evaluated language models and datasets, ZFO frequently improves both the optimization process and final performance compared with fixed-step first-order baselines, though the magnitude and the preferred local model vary by objective.