Zeroth-Order Optimization Is Underexplored, Not Underpowered: Training Deep Models Without Backpropagation

Forum topic · 小凯 · 2026-05-19

Summary

A Chinese tech forum post discusses a recent position paper arguing that zeroth-order optimization (ZOO) in deep learning is underrated. ZOO estimates gradients from function-value differences alone—perturbing parameters and observing loss changes—without backpropagation. This matters when optimizing closed-source models via API calls or when backward-pass memory exceeds hardware limits. ZOO has long been dismissed due to high variance and query costs, but the paper by Liu, Lang, Pal and colleagues contends the real problem is short-sighted research design: current ZOO methods are full-space, element-wise, and estimator-centric. The authors outline six underexplored directions, including subspace and spectral ZOO with graceful query scaling in dimension, and exploiting forward-only computation for simpler pipeline parallelism, less communication, and lower memory. The post also notes open questions: no new algorithms are provided, subspace selection is unsolved, query efficiency on 100B+ parameter models remains unverified, and the memory-versus-query-cost tradeoff lacks systematic quantification.

Deep learning is built on one premise: you can compute gradients—backpropagation for every parameter. But some scenarios make this impossible. You might be optimizing a closed-source model accessible only through API calls, or the model may be so large that backpropagation exceeds your hardware's memory limits.

Zeroth-order optimization (ZOO) skips backpropagation entirely, estimating gradients from differences in function values: add a small perturbation to the parameters, observe how the loss changes, and step in the direction that reduces it. Two theoretical flaws are repeatedly cited—high estimator variance and excessive query counts—which is why ZOO has long been considered impractical.

Liu, Lang, Pal and their team take the opposite stance: ZOO is underexplored rather than underpowered, and its poor reputation stems from short-sighted development. Specifically, existing research has made ZOO "full-space, element-wise, and estimator-centric"—perturbing and differencing each parameter dimension independently. This is not the only approach, and arguably not the best one.

The paper identifies six directions awaiting development:

Subspace and spectral views of ZOO can be interpreted as denoising, with query counts that grow gracefully with dimension.
Forward-only computation is itself a systems advantage—no backward pass means simpler pipeline parallelism, less communication, and lower memory footprint.
Benchmarking problems: part of why ZOO performs poorly in existing evaluations is that the benchmarks fail to separate task complexity from optimizer capability.

Open questions: the paper provides no new algorithms—only directions. How should a suitable subspace be selected automatically in practical subspace ZOO? On genuinely large models (100B+ parameters), can query counts be kept within acceptable bounds? The tradeoff curve between memory savings and query cost has never been systematically quantified.

---

References

1. Liu, S., Lang, Y., Pal, S., et al. (2026). *Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered*. arXiv:2605.15622 [cs.LG]. 2. Malladi, S., et al. (2023). *Fine-Tuning Language Models with Just Forward Passes*. NeurIPS. 3. Spall, J. C. (1998). *An Overview of the Simultaneous Perturbation Method for Efficient Optimization*. Johns Hopkins APL Technical Digest.

Zeroth-Order Optimization Is Underexplored, Not Underpowered: Training Deep Models Without Backpropagation

Summary

Tags