Forgetting in Batches: What Quantization Reveals About the Uneven Structure Inside LLMs

Here's a counterintuitive fact first.

Llama-3's 70B version needs at least 140GB of VRAM at 16-bit precision. Compress all weights to 4-bit, and it drops to 35GB—small enough to fit on a gaming laptop. The number of distinct values each parameter can express falls from tens of thousands to sixteen, a three-quarter cut in precision.

By intuition, the model should get "three-quarters dumber." In reality? For everyday conversation, writing, and translation, most people can't tell the difference.

This isn't magic, nor is it "models aren't that complex after all." Quite the opposite: it shows the model has an extremely uneven internal structure. Let's take this apart, and look at the load-bearing map it reveals.

Knowledge doesn't live in a single parameter

A calculator: get one digit slightly wrong, and the result is wrong. Neural networks aren't that kind of machine. Their knowledge is spread across hundreds of millions or billions of parameters—no single parameter is individually responsible for "remembering" something.

So when you cut the precision of many parameters, the errors cancel each other out. This one tilts a bit, that one tilts a bit, and the combined output barely changes. It's like a choir: a few singers go slightly off-pitch, but it's still the same song. Redundancy is the capital that quantization spends.

But "unimportant" doesn't mean "fair game"

Redundancy is capital, but not unlimited capital. Individual parameters don't matter much; the structure *between* them does. Some channels are like load-bearing walls: invisible in normal times, but hit one and the whole building shakes.

There's experimental evidence. The AWQ paper from Song Han's group at MIT (MLSys 2024 Best Paper) includes a comparison table for 3-bit compression of OPT-13B. First, the ruler in that table: perplexity, roughly a measure of how hesitant the model is about the next token—lower is better. Full-precision OPT-13B scores 10.13. Compress everything to 3-bit and it jumps to 46.04—the model is basically ruined.

Now keep just one-in-a-thousand weight channels at full precision while compressing the rest to 3-bit. How do you pick that one-in-a-thousand?

Pick by weight magnitude. The channels with big numbers should be "most important," right? Result: perplexity 46.07—nearly identical to not picking at all. The protection was wasted.

Pick by activation distribution: look at which channels actually get used on every real input. Same one-in-a-thousand budget: perplexity 10.51, almost back to 10.13.

Same budget, same quantity—choosing whom decides life or death. "What matters is not which number is big, but which channel is actually doing work." This is AWQ's core finding, and it's a blind spot in many people's intuition about model internals.

Another route: not picking whom to protect, but choosing where errors go

GPTQ takes a different approach. When quantizing each weight, it doesn't leave the quantization error in place—it lets the not-yet-quantized neighbors compensate. How much to compensate, and in which direction, is computed from second-order information—think of it as knowing how nudging one weight triggers chain reactions among the others.

The result: a 175-billion-parameter model compressed to 3-bit in about four GPU hours with negligible accuracy loss; OPT-175B, which originally needed five 80GB GPUs, now runs on a single A100.

GPTQ and AWQ use completely different strategies, but share one principle: errors are not scattered evenly. One pushes errors where they cancel out; the other leaves precision where errors can't be tolerated.

The model really has squatters that won't budge

The most direct evidence of this unevenness comes from LLM.int8(). Researchers found that large models contain a small set of feature dimensions with magnitudes dozens of times larger than others, systematically appearing at fixed positions—starting at roughly the 6.7-billion-parameter scale. Nudge these dimensions and the attention output collapses.

The fix is crude but effective: keep this small handful of dimensions in 16-bit, and run the other 99.9% of multiplications in 8-bit. A 175-billion-parameter model with zero accuracy loss.

Three methods, three lines of evidence, all pointing to the same thing: model internals are not a uniform porridge. The vast majority of locations are forgiving; a tiny few are squatters you can't evict.

1.58-bit is a different story

So how low can you compress? There's a beautiful lower-bound case: BitNet b1.58. Each weight takes only three values—-1, 0, 1—averaging 1.58 bits, far harsher than 4-bit. It matches full-precision models of the same size and training budget on perplexity and task performance.

But note the premise: it grew up in a ternary world from the start—it isn't a trained model compressed afterward. This answers a hidden question: how much compression a model can withstand depends on whether precision was a budget planned during training, or an asset confiscated after the fact.

Post-hoc compression down to 2-bit is a different kind of difficulty, requiring more delicate surgery like codebooks and rotations to stand up. Same bit count, different origins, different fates.

So where does it first start forgetting?

Back to the opening question: at what point does removing precision make the model truly forget what it learned?

A 2025 systematic study (Quantization Meets Reasoning) specifically tested quantization's impact on mathematical reasoning. The conclusion: there is no single universal "forgetting line." Everyday tasks are nearly lossless at 4-bit, but math reasoning degrades first—up to 32.39% on Llama-3, averaging 11.31%, with numerical computation and reasoning planning hit hardest. Perplexity can still look great while some capabilities have already quietly exited.

Even more interesting is the recovery experiment: fine-tuning for 3 minutes on just 545 task samples basically restores reasoning ability to full-precision levels. The model didn't "forget"—the path to answers just got rougher. The knowledge is still there; the signposts are worn.

So the most interesting byproduct of quantization isn't how much VRAM it saves. It's currently the cheapest CT scanner available: each compression step produces a scan—showing which capabilities live shallow, which live deep, and which one-in-a-thousand channels are the load-bearing walls of the entire building.

In the next 4-bit weight table, the channels kept on a 16-bit life-support are the model's true structural blueprint. What engineers do is plain and simple: don't ask where the value is—ask where you must not touch.

---

Sources: AWQ (MLSys 2024, arXiv 2306.00978, comparison data from its Table 1, w3-g128 setting); GPTQ (ICLR 2023, arXiv 2210.17323); LLM.int8() (arXiv 2208.07339); BitNet b1.58 (arXiv 2402.17764); Quantization Meets Reasoning (arXiv 2501.03035).