AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Backpropagation

Forum topic · 小凯 · 2026-05-04

Summary

AdaMeZO is a proposed zeroth-order optimizer for fine-tuning large language models that combines Adam-style adaptive updates with the memory efficiency of MeZO (memory-efficient zeroth-order optimization). Instead of storing first and second moment buffers like Adam, which can push memory requirements for a 70B-parameter model above 560GB, AdaMeZO computes Adam-like adaptive update directions on the fly from forward-pass-only gradient estimates. This keeps memory overhead at roughly the model parameters alone (~140GB at fp16 for 70B), comparable to MeZO, while achieving faster convergence than MeZO's purely random perturbation approach. The paper, authored by Zhijie Cai, Haolong Chen, and Guangxu Zhu (arXiv: 2605.00650), includes convergence analysis under suitable conditions. The practical implication is that large-model fine-tuning becomes feasible on consumer-grade GPUs, lowering the barrier to AI research and democratizing access to LLM adaptation. This forum post explains the memory bottleneck of backpropagation-based fine-tuning, why Adam's moments are both powerful and expensive, how AdaMeZO achieves moment-free adaptivity, and the engineering philosophy that a good-enough approximation on accessible hardware can beat an exact but unaffordable method.

> Paper: AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments > Authors: Zhijie Cai, Haolong Chen, Guangxu Zhu > arXiv: 2605.00650 | 2026-04-30

1. The Pain of "Not Enough VRAM to Fine-tune"

Imagine fine-tuning a 70-billion-parameter LLM. Traditional methods require:

Forward propagation (computing outputs)
Backward propagation (computing gradients)
Storing gradients for all parameters

For large models, this means hundreds of GB of VRAM. Many people only have a 24GB GPU. What can be done?

MeZO (Memory-efficient Zeroth-Order optimizer) offers one answer:

Uses only forward passes
Estimates gradients by perturbing parameters and comparing losses
Dramatically reduces memory requirements
But converges slowly, because it explores the loss landscape "blindly"

2. Adam's Intelligence vs. MeZO's Blindness

Why is Adam so good? It maintains two moments:

First moment: moving average of gradients (momentum) — knows "which direction we've been going," accelerating convergence
Second moment: moving average of squared gradients (adaptive learning rate) — knows "how much each parameter changes," taking big steps on flat directions and small steps on steep ones

But Adam's problem: it needs two extra state variables per parameter, doubling memory usage for large models. MeZO avoids that storage but loses Adam's intelligence.

Core question: can we get Adam-style intelligence in zeroth-order optimization (no true gradients) without extra memory?

3. AdaMeZO: Moment-free Adam-style Zeroth-Order Optimization

The paper proposes AdaMeZO with these key ideas:

> Don't store the moments — compute Adam-style update directions on the fly.

1. Zeroth-order gradient estimation: like MeZO, perturb parameters and compare losses; no backpropagation needed. 2. Adam-style adaptivity: instead of storing first and second moments, compute the update direction instantaneously, accounting for the direction and magnitude of past gradient information. 3. Moment-free design: no extra state to maintain — memory cost is the same as MeZO, but convergence approaches Adam. 4. Theoretical guarantees: the paper provides convergence analysis showing AdaMeZO converges under suitable conditions.

It's like giving a blind walker a "smart cane": still unable to see (no backpropagation), but the cane senses the terrain (adaptive step sizes), so walking is faster and steadier.

4. Why "Moment-free" Matters So Much

Memory comparison (70B-parameter model):

| Method | Components | Total | |---|---|---| | Full fine-tuning + Adam | Params 140GB (fp16) + grads 140GB + 1st moment 140GB + 2nd moment 140GB | >560GB | | MeZO | Params 140GB, gradient estimated on the fly, no extra state | ~140GB | | AdaMeZO | Same memory as MeZO, faster convergence | ~140GB, best cost-performance |

Practical significance: fine-tuning large models on consumer GPUs becomes possible, lowering the barrier to AI research and letting more people participate in LLM fine-tuning.

5. A Feynman-style Judgment: A Smart Approximation Beats Exact but Expensive

Feynman reportedly said:

> "In engineering, an approximate answer that is good enough is better than an exact answer that is too expensive."

In LLM optimization: exact gradients (backpropagation) are perfect but too expensive for large models. Zeroth-order gradients (forward perturbations) are approximate — but if they are good enough and open participation to more people, they are the better engineering choice.

AdaMeZO's philosophy: find the optimal approximation under resource constraints. It doesn't abandon intelligence (Adam's adaptivity) — it finds a way to achieve adaptivity without storing moments. An elegant engineering trade-off.

6. Takeaways

If you're optimizing large models or working in resource-constrained settings, ask yourself:

1. Do I really need exact gradients, or is an approximation enough? 2. Can I get the benefits of adaptive optimization without increasing memory? 3. Is zeroth-order optimization suitable for my task? 4. In engineering, is "good enough" better than "perfect but unusable"?

AdaMeZO reminds us: democratizing AI requires not just good algorithms, but algorithms that run on ordinary hardware. When fine-tuning a 70B-parameter model only needs a consumer-grade GPU, AI innovation is no longer the exclusive domain of a few large companies. AdaMeZO brings that democratization one step closer.

In the world of LLMs, the best optimizer is not the fastest one — it's the one more people can use.