The Illusion of Quantization: What Do Models Really Lose When Precision Disappears?

A Cup of Coffee That Lost Its Flavor

You walk into your usual café and order a pour-over Yirgacheffe. The barista says: "Today the beans were roasted a bit lighter, the water temperature is one degree higher, and the grind is half a notch finer." You take a sip — it's still coffee, still good. But a regular would frown: the floral notes are gone, the finish is short, and that bright citrus acidity has turned into a dull sweetness.

If you judge coffee only by "is it coffee?" and "does it taste good?", this cup is nearly equivalent to yesterday's. But what defines a coffee's soul is the molecular composition hidden in the details.

Large model quantization faces the same dilemma.

When you compress a 7B-parameter LLM from 16-bit to 4-bit, the size shrinks to a quarter, inference speed doubles, and the accuracy loss "looks like" only 1-2 percentage points. Everything seems fine. But in July 2026, three researchers from the University of Manitoba, Red River College, and the University of Central Florida — Baha Rababah, Cuneyt Gurcan Akcora, and Carson K. Leung — published a paper titled *The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs*, using a new metric to tear away this disguise of "equivalency."

The paper's core finding can be summarized in one sentence: **a quantized model, even if its accuracy and perplexity look unchanged, has quietly swapped personalities in terms of *which questions it gets right*.

The Old Friends of Quantization Practitioners: Accuracy and Perplexity

First, let's clarify what quantization does.

LLM parameters are essentially floating-point numbers — usually 16-bit floats (FP16). Quantization compresses these to lower precision: 8-bit, 4-bit, even 2-bit. The benefits are obvious: a 7B model shrinks from 14GB to 3.5GB, runnable on consumer GPUs; inference speeds up 2-3x; energy consumption drops. The cost is precision loss.

The industry has evaluated this cost the same way for decades:

• Perplexity (PPL): how "confused" the model is by a piece of text — lower is better. Measures language modeling ability.
• Downstream task accuracy**: how many questions the model answers correctly on benchmarks like HellaSwag, Winogrande, and ARC.

These two metrics are like coffee's "is it coffee" and "does it taste good." They tell you whether the model is still usable, but not how much the model has changed in *how* it is used.

The paper ran a very direct experiment. They quantized four models — Llama-3.2-3B, Vicuna-7B, Mistral-7B, Llama-3.1-8B — using llama.cpp legacy quantization (Q8_0, Q5_0, Q4_0) and K-quantization (Q6_K down to Q2_K), compressing from 8 bits all the way to 2.

Results on perplexity: Llama-3.2-3B's baseline PPL is 2.300, Q8_0 is 2.303, Q5_0 actually drops to 2.170, and Q3_K drops to 1.967. Perplexity didn't rise — it fell. Quantization made the model "less confused"?

But on accuracy: baseline 55.5%, Q8_0 53.4%, Q3_K 51.5%. It is indeed declining, but gently — seemingly no big deal.

This is where the "illusion of equivalency" comes from: the surface metrics are so stable that you assume nothing has changed.

Correctness Agreement: A Simple but Sharp New Metric

The paper's real contribution is a metric called Correctness Agreement (CA). Its definition is surprisingly simple:

> For each question where the base model is correct, how many of those does the quantized model also answer correctly, as a fraction of all questions?

The formula is:

\[\text{CA}(c; \theta, \mathcal{D}) = \frac{1}{M} \sum_{m=1}^{M} \mathbb{1}[z_m = 1 \land z_m^{(c)} = 1]\]

where \(z_m\) is the base model's correctness label on question \(m\) (1 right, 0 wrong), and \(z_m^{(c)}\) is the quantized model's label.

This metric has a clever property: it is always less than or equal to the minimum of the two models' accuracies. In other words, even if the quantized model's accuracy equals the base model's, CA can be far lower — because the questions they get right may be different ones.

This is the key: two models can have identical accuracy while getting a completely different set of questions right.

The experimental data is unsettling. Take Llama-3.2-3B:

| Quantization | Accuracy | CA | |---------|--------|-----| | Base | 55.5% | — | | Q8_0 | 53.4% | 41.4% | | Q6_K | 52.9% | 41.1% | | Q5_0 | 52.6% | 41.0% | | Q4_K | 52.1% | 40.9% | | Q3_K | 51.5% | 39.9% |

Looking at accuracy: Q8_0 is only 2.1 points below baseline, seemingly acceptable. But looking at CA: only 41.4%. This means about 14 percentage points (roughly a quarter of the correct answers) of the base model's 55.5% correct answers became wrong after quantization. Meanwhile, the quantized model also answers some questions the base model got wrong, so accuracy looks barely changed.

The model didn't "get a bit worse" — it "swapped in a different batch of correct answers."

This finding has direct implications for real applications. Imagine using an LLM for medical diagnostic assistance: the base model answers 55 of 100 cases correctly; after quantization, 53. Looks like a difference of 2. But if 14 of those 53 are different from the base model's — meaning the quantized model gives different correct diagnoses on 14 cases while getting 14 cases wrong that the base model answered correctly — would you dare deploy it?

Quantization Is Not a Uniform Blade: Q/K Are More Fragile Than V/O

The paper's second major finding comes from structural analysis of attention weights.

A Transformer's self-attention layer has four projection matrices: Query (Q), Key (K), Value (V), and Output (O). Q determines "what I'm looking for," K determines "what I can offer," V is the actual information content, and O is the output transformation.

The researchers computed all the statistical changes after quantization (mean, standard deviation, skewness, kurtosis) and distributional differences (cosine similarity, Euclidean distance, KS statistic, KL divergence) for these four matrices. A clear pattern emerged:

Q and K projections are the most sensitive to quantization; V and O are relatively stable.

Specifically, under Q3_K and Q2_K (3-bit and 2-bit quantization):

• Q and K skewness fluctuates violently, kurtosis collapses, and means shift noticeably
• V retains its low-kurtosis character with relatively intact structure
• O has the highest kurtosis but shows no obvious degradation until the lowest bit widths

The KL divergence data is even more intuitive: as bit width decreases, Q and K weight distributions drift further from the base model, while V and O change far more gently.

There's a natural analogy here. Imagine attention as a large ballroom dance party:

• Q is each guest's "person-finding radar" — "I'm looking for someone who knows machine learning"
• K is the "identity tag" on everyone's chest — "I work on NLP"
• V is what guests can actually talk about
• O is the social notes compiled after the party

Quantization is like dimming the lights. Q and K depend on fine-grained matching — the radar must distinguish "knows machine learning" from "knows machine repair," the tags must distinguish "NLP" from "NLP papers." Once the lights dim, these fine-grained matches fail first. V and O are more like recording what has already happened, less sensitive to lighting.

The paper thus offers an engineering suggestion: future quantization methods should allocate precision adaptively — giving Q and K more bits and V and O fewer bits. This is more sensible than one-size-fits-all uniform quantization.

Three Regimes of Quantization: Safe, Degraded, Collapsed

By scanning the full range from 8-bit to 2-bit, the paper identifies three clear quantization regimes:

Safe zone (Q8_0 to Q4_K): statistical metrics barely change, perplexity is stable, but CA is already below the accuracy ceiling — behavioral drift exists, just masked by aggregate metrics.

Degradation zone (Q3_K): skewness starts to surge, kurtosis starts to collapse, KL divergence rises sharply. CA declines further. This is the turning point where structural damage begins to exceed functional compensation.

Collapse zone (Q2_K): all metrics deteriorate across the board. The weight distribution is unrecognizable compared to the base model. The model is functionally untrustworthy.

This three-way classification is practical for practitioners. Today people choose quantization schemes mainly by "how much accuracy drop is acceptable." But the paper's data shows Q4_K's accuracy may be only 0.5 points below Q5_K's, while CA may differ by several points. You think you lost 0.5% accuracy; in reality you lost a quarter of the model's "decision identity."

Perplexity Can Lie

The paper has another finding worth highlighting: there is a disconnect between perplexity and behavioral consistency.

Llama-3.2-3B's perplexity under Q3_K is 1.967 — lower than the baseline's 2.300. By traditional standards, this is a "better" model. But its CA is only 39.9%, among the lowest of all schemes.

What does this mean? Perplexity measures the model's probability estimates over text; low PPL means the model is "unsurprised" by the test text. But this "unsurprise" may come from a smoothing effect introduced by quantization — extreme probabilities are compressed, all predictions become more "middling," and overall PPL drops. Yet the model's judgment on specific questions has already changed.

A model that is "unsurprised by everything" is not necessarily a model with accurate judgment. It may simply have lost its sensitivity to fine distinctions.

This reminds me of a psychology experiment: subjects shown a series of face photos judge "is this person trustworthy." If you give the subjects blurry glasses, they might say "they're all fine" — the "variance" of overall judgments drops, but judgments on each individual face are already unreliable. Low perplexity is like this "blurry glasses effect."

Engineering Insights: What This Means

The paper has several direct practical implications for AI practitioners:

1. Evaluating quantized models cannot rely on aggregate metrics alone. Accuracy and perplexity are necessary but insufficient. If your application is sensitive to *which questions are answered correctly* (e.g., healthcare, legal, safety moderation), add CA or similar behavioral consistency metrics.

2. Quantization scheme selection should look at the CA inflection point, not the accuracy inflection point. The paper's data shows Q4_K is the safe upper bound and Q3_K begins to degrade. This is more reliable than "stop when accuracy drops 2%."

3. Adaptive quantization has a clear direction. Q and K need higher precision; V and O can be compressed more aggressively. This provides empirical grounding for future mixed-precision quantization.

4. Low perplexity does not mean a good model. If you see PPL drop after quantization, don't celebrate too early — check behavioral consistency.

5. Quantization is not "lossless compression." Even at 8-bit, CA is already clearly below the accuracy ceiling. Every quantization changes the model's decision identity — it's only a matter of degree.

A Deeper Reflection: Identity and Capability

This paper raises a philosophical question: what is a model's "identity"?

The traditional view is that a model's identity is its capabilities — how accurate it is, what it knows. As long as these are unchanged, the model is still the same model.

But the paper offers another perspective: a model's identity lies in its judgments on specific cases. Two models with the same accuracy that get different questions right are two different "decision-makers."

This matches the experience of human experts. Two doctors may both have 90% diagnostic accuracy, but if their judgments on specific cases frequently diverge, you would say they are "doctors of different schools," not "equivalent doctors." A quantized model is essentially "a doctor of a different school" — its overall level may be similar, but its judgments on specific cases have changed.

For AI systems that demand reproducibility, auditability, and accountability, this "identity drift" is a severely underestimated risk. You deploy a quantized version based on the base model's test report, assuming behavioral consistency. But on 14% of cases, they give different judgments. Some of these differences may not surface on the test set, yet suddenly appear in the long-tail distribution of the real world.

The cost of quantization is not "getting a bit dumber," but "becoming a different decision-maker." That other decision-maker may be equally smart — but it is smart in a different way.

Paper Information

• Title: The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs
• Authors: Baha Rababah, Cuneyt Gurcan Akcora, Carson K. Leung
• Institutions: University of Manitoba, Red River College, University of Central Florida
• arXiv: 2607.08734
• Quantization framework: llama.cpp
• Models tested: Llama-3.2-3B, Vicuna-7B, Mistral-7B, Llama-3.1-8B
• Benchmarks: WikiText-2, C4, HellaSwag, Winogrande, ARC

*Quantization is an engineering reality, not a choice to be avoided. But this paper reminds us: every precision compression changes the model's decision-making soul. Next time you see a report claiming "accuracy only drops 1% after quantization," remember to ask: what about the CA?*