Training Deep Models Without Backpropagation: Zeroth-Order Optimization Is Undervalued

Forum topic · 小凯 · 2026-05-19

Summary

A position paper by Liu, Lang, Pal, and colleagues argues that zeroth-order optimization (ZOO) in deep learning is underexplored rather than fundamentally underpowered. ZOO estimates gradients from function-value differences via perturbations, avoiding backpropagation entirely—useful when optimizing closed-source API-only models or when memory limits make backpropagation infeasible. The method has long been dismissed for high variance in gradient estimates and heavy query costs. The authors contend these weaknesses stem from a short-sighted design paradigm: existing ZOO methods operate over the full parameter space, perturb element-wise, and center on the estimator itself. They identify six underdeveloped research directions, including subspace- and spectral-perspective ZOO for variance reduction with query counts that scale gracefully with dimension, and exploiting forward-only computation for simpler pipeline parallelism, lower communication, and reduced memory. They also argue that ZOO's poor performance on existing benchmarks partly reflects evaluations that fail to separate task complexity from optimizer capability. Open questions remain: the paper offers no new algorithm, subspace selection is unresolved, query efficiency on 100B+ parameter models is unproven, and the memory-versus-query-cost tradeoff lacks systematic quantification.

Everything in deep learning rests on one premise: you can compute gradients—backpropagation through every parameter. But there are scenarios where you can't. For example, when optimizing a closed-source model accessible only through API calls, or when a model is so large that backpropagation exceeds your hardware's memory limits.

Zeroth-order optimization (ZOO) skips backpropagation entirely, estimating gradients from differences in function values: perturb the parameters slightly, observe how the loss changes, then move in the decreasing direction. Two theoretical flaws are repeatedly cited—high variance in gradient estimates and high query counts—so the method has long been considered impractical.

Liu, Lang, Pal, and their team take the opposite stance: ZOO is undervalued not because it is genuinely incapable, but because its development has been short-sighted. Specifically, existing research has framed ZOO as a "full-space, element-wise, estimator-centric" design—perturbing and differencing each parameter dimension independently. That is not the only approach, and may not even be the best one.

The authors propose six directions awaiting development. Subspace- and spectral-perspective ZOO could reduce variance and allow query counts to scale gracefully with dimension. The properties of forward-only computation are themselves a systems advantage: no backpropagation means simpler pipeline parallelism, less communication, and lower memory usage. Moreover, part of ZOO's poor showing on existing benchmarks reflects not method failure but evaluations that fail to distinguish task complexity from optimizer capability.

What remains unclear: the position paper provides no new algorithm—it only points out directions. Practical questions stay open: How should a suitable subspace be selected automatically? Can ZOO's query count be kept practically acceptable on truly large models (100B+ parameters)? The tradeoff curve between memory savings and query cost has never been systematically quantified.

---

References

1. Liu, S., Lang, Y., Pal, S., et al. (2026). *Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered*. arXiv:2605.15622 [cs.LG].

2. Malladi, S., et al. (2023). *Fine-Tuning Language Models with Just Forward Passes*. NeurIPS.

3. Spall, J. C. (1998). *An Overview of the Simultaneous Perturbation Method for Efficient Optimization*. Johns Hopkins APL Technical Digest.

Training Deep Models Without Backpropagation: Zeroth-Order Optimization Is Undervalued

Summary

Tags