Cutting Optimizer State 16x: Ternary Sparse Update Lets a Single GPU Fine-Tune 32B Models

Where does the memory go when you fine-tune a large model?

The answer is neither the model weights nor the gradients. On an 80 GB H100, weights and gradients each take a dozen or so GB; the remaining fifty to sixty GB are all optimizer state. AdamW keeps two extra numbers per parameter—a first moment and a second moment. Add the low-precision master weight copy, and each parameter occupies four entries during training.

Scale the model up and those entries blow past the card. OPT-13B under AdamW8bit peaks at 80.6 GB; plain AdamW would push that even higher.

TACO, submitted to arXiv on October 1, takes an approach simple almost to the point of brute force: for each column of every parameter matrix, update only the element with the largest absolute value in that column—and update only its sign.

Three Roads to Saving Memory

The paper splits existing methods into three categories, each with its own pitfall.

• Compressing optimizer state: 8-bit Adam belongs here—trading precision for space.
• Abandoning first-order gradients: MeZO and ZO-Muon take the zeroth-order route, estimating directions by perturbation-based loss re-evaluation.
• Changing update geometry while keeping dense state: GaLore and APOLLO project gradients and state into low-dimensional subspaces. Muon, the recent newcomer, compresses optimizer memory with matrix-valued updates, but it changes the geometry and loses accuracy when fine-tuning AdamW-pretrained models.

TACO claims Muon's road, but pushes it one step further. Muon performs matrix-valued updates within the steepest-descent direction given by operator norms; TACO goes straight to the endpoint: it exactly solves the steepest-descent direction under the dimension-normalized 1→1 operator norm, which lands as "pick one element per column, take the sign."

174x and 2.9x

The headline numbers (OPT-13B, versus AdamW8bit):

| Metric | AdamW8bit | TACO | Ratio | |---|---|---|---| | Resident optimizer state | 27.7 GB | 0.16 GB | 174x | | Peak training memory | 80.6 GB | 27.5 GB | 2.9x |

27.7 divided by 0.16 gives 173.1; the paper writes 174x, the difference coming from more precise internal values.

The 0.16 GB figure deserves a pause. Compressing 27.7 GB of state into 0.16 GB is like packing an entire room into one suitcase. The method is not complicated: keep only a few low-precision gradient components per column, combined with FP8 sparse gradient history and on-the-fly gradient processing, dropping resident state from O(mn) to O(n).

The saved space converts directly into model size. Full-parameter fine-tuning of 30–32B models on a single 80 GB H100 is the paper's live demonstration, run on OPT-30B and Qwen3-32B.

The cost column reads "comparable accuracy and runtime." That is the paper's own wording, not a third-party evaluation.

Where the Geometry Comes From

TACO's skeleton is steepest descent under operator norms.

Bernstein and Newhouse's 2024 work formulated matrix optimization as steepest descent under operator norms and characterized the family of p-norms from 1 to infinity; Pethick et al.'s 2025 Scion provided a stochastic implementation. TACO continues down this path, landing on the dimension-normalized 1→1 norm.

Under the 1→1 norm, the steepest-descent direction has a closed-form solution: the sign of the largest-magnitude element in each column. So the ternary sparsity is not post-hoc pruning forced onto the updates—it is what the geometry itself looks like. The paper takes care to distinguish these two things.

The cost is that updates become extremely sparse: each element waits many steps for its turn. This is where the cost comes from, and why the paper runs dedicated ablations: EMA decay coefficient of the gradient history, heavy-tail threshold, update sparsity, and optimizer step budget—ablations on all four.

On convergence, the paper proves that vanilla TACO reaches an eps-stationary point in O(eps^-2) steps under standard smoothness assumptions. There is also a result I find more interesting than the headline: in its theoretical setting, TACO shares continuation geometry and limiting solutions with an Adam surrogate—a property that distinguishes it from Muon while satisfying standard muP spectral scaling (under balanced winner counts).

Cross-family transfer was also tested: OPT, Qwen3, Pythia, plus Llama and Mistral all work.

Why This Road Relates to Coding Agents

At first glance this is an optimizer-for-training story, unrelated to AI coding.

But it makes sense in a 2026 context. Frontier practice for coding agents has shifted toward test-time compute and post-training, which is a memory bill: pushing a model past some benchmark's passing bar at a scale a lab can afford, the bottleneck is often not compute but the tens of GB the optimizer state occupies. Kimi K2.7 Code doing a round of post-training with rank-32 LoRA and GSPO, NVIDIA using SoL-Pi for test-driven automated optimization to save tokens—all are counting money from the same pocket.

The significance of full-parameter fine-tuning of 32B on a single card is the same. It turns "we have no H100 cluster, so we can only do LoRA" from a funding question into a memory question.

The paper is 24 pages, 7 figures, 10 tables; code at github.com/Jichao2357/TACO_optimizer. Preprint, not peer-reviewed.

Three Caveats You Should Not Skip

First, "comparable accuracy and runtime" is the paper's own claim. It provides no complete per-task comparison table and no third-party replication.

Second, the 2.9x is peak training memory, not end-to-end cost. The 174x is resident optimizer state, not a reduction of total training overhead to 1/174th. The training speed bottleneck is not necessarily memory bandwidth, and the paper does not fully reconcile these.

Third, all experiments sit in a post-Muon context. Muon itself is a recent development; TACO stands on its shoulders and pushes one step further. Idea half-life in the optimizer field is short, and whether the 174x number survives the next hardware architecture shift is not visible now.

That said, the methodological contribution is solid. Ternary sparsity is not an ad-hoc heuristic—it is a closed-form solution derived from operator-norm steepest descent. Things of this kind usually outlive hardware generations.

---

Sources: arXiv:2610.02199 (Jichao Jiang, Cristian McGee, El Houcine Bergou, Hanqin Cai, Aritra Dutta; University of Central Florida and Mohammed VI Polytechnic University, Morocco), v1 submitted 2026-10-01 17:59 UTC, 24 pages, 7 figures, 10 tables, code at github.com/Jichao2357/TACO_optimizer. Theoretical groundwork: Bernstein & Newhouse 2024, Pethick et al. 2025.

Caveats: All comparison figures come from the paper's own experiments; no third-party replication found. "Comparable accuracy and runtime" is the paper's exact wording. 174x is rounded by the paper; 27.7 GB / 0.16 GB gives 173.1. arXiv v1 preprint, not peer-reviewed.