AdaMeZO: An Adam-Style Zeroth-Order Optimizer for LLM Fine-Tuning Without Maintaining Moments

Overview

This forum post discusses AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments, by Zhijie Cai, Haolong Chen, and Guangxu Zhu (arXiv:2605.00650, dated 2026-04-30).

The Problem: Fine-Tuning Large Models Without Enough Memory

Fine-tuning a 70-billion-parameter LLM with traditional methods requires forward passes, backpropagation, and stored gradients — often hundreds of GB of GPU memory. Many practitioners only have a 24GB consumer GPU.

MeZO (Memory-efficient Zeroth-Order optimizer) offers an alternative:

• Uses only forward passes
• Estimates gradients by perturbing parameters and comparing losses
• Drastically reduces memory requirements
• But converges slowly because it explores the loss landscape "blindly"

Adam's Intelligence vs. MeZO's Blindness

Adam works well because it maintains two moments:

• First moment (momentum): a moving average of gradients, knowing "which direction we've been heading," accelerating convergence
• Second moment (adaptive learning rate): a moving average of squared gradients, taking big steps on flat directions and small steps on steep ones

But Adam requires two extra state variables per parameter, roughly doubling memory for large models. MeZO avoids storing state but loses Adam's adaptivity.

Core question: Can we get Adam-style intelligence in zeroth-order optimization (no true gradients) without extra memory?

AdaMeZO's Approach

The key idea: compute Adam-style update directions on the fly, without storing moments.

1. Zeroth-order gradient estimation — like MeZO, estimates gradients via forward passes and parameter perturbations; no backpropagation needed 2. Adam-style adaptivity — instead of storing first/second moments, computes the update direction immediately, accounting for historical gradient direction and magnitude 3. Moment-free design — the key breakthrough: no extra persistent state, so memory overhead matches MeZO while convergence approaches Adam 4. Theoretical guarantees — the paper provides convergence analysis showing AdaMeZO converges under suitable conditions

It's like giving a blind walker a "smart cane": still no sight (no backprop), but the cane senses terrain (adaptive step sizes), making walking faster and steadier.

Why "Moment-Free" Matters: Memory Comparison (70B model)

| Method | Memory breakdown | Total | |---|---|---| | Full fine-tuning + Adam | 140GB params (fp16) + 140GB gradients + 140GB first moment + 140GB second moment | >560GB | | MeZO | 140GB params, gradients computed on the fly, no extra state | ~140GB | | AdaMeZO | Same memory as MeZO, faster convergence | ~140GB, best value |

Practical implications:

• Fine-tuning large models on consumer GPUs becomes feasible
• Lowers the barrier to AI research and applications
• More people can participate in LLM fine-tuning

Takeaways

If you're optimizing large models under resource constraints, ask:

1. Do I really need exact gradients, or are approximations enough? 2. Can I get adaptive optimization benefits without extra memory? 3. Is zeroth-order optimization suitable for my task? 4. In engineering, is "good enough" better than "perfect but unusable"?

AdaMeZO reminds us that democratizing AI requires not just good algorithms, but algorithms that run on ordinary hardware. When fine-tuning a 70B model needs only one consumer GPU, AI innovation is no longer the privilege of a few big companies. The best optimizer isn't the fastest — it's the one more people can use.

---

Source: arXiv:2605.00650 — AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments