The Illusion of Quantization: What Do LLMs Really Lose When Precision Disappears?

A Cup of Coffee That Lost Its Flavor

You walk into your regular coffee shop and order a pour-over Yirgacheffe. The barista says: "Today's beans were roasted a bit lighter, the water temperature is one degree higher, and the grind is half a step finer." You take a sip — it's still coffee, still good. But a regular customer would frown: the floral aroma is gone, the finish is shorter, and that bright citrus acidity has dulled into flat sweetness.

If you judge the coffee only by "is it coffee" and "does it taste good," this cup is nearly equivalent to yesterday's. But what really defines a cup of coffee's soul is its molecular composition hidden in the details.

Large model quantization faces the same dilemma.

When you compress a 7B-parameter LLM from 16 bits down to 4, the size shrinks to a quarter, inference speed doubles, and the accuracy loss "looks like" only 1-2 percentage points. Everything seems fine. But in July 2026, three researchers from the University of Manitoba, Red River College, and the University of Central Florida — Baha Rababah, Cuneyt Gurcan Akcora, and Carson K. Leung — published a paper, *The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs*, using a new metric to tear off this mask of "equivalence."

The paper's core finding can be summarized in one sentence: **a quantized model, even when accuracy and perplexity look unchanged, has quietly swapped personalities in terms of *which questions it answers correctly*.

The Old Friends of Quantization Practitioners: Accuracy and Perplexity

First, let's clarify what quantization does.

LLM parameters are essentially floating-point numbers — usually 16-bit floats (FP16). Quantization compresses these into lower precision: 8-bit, 4-bit, even 2-bit. The benefits are obvious: a 7B model shrinks from 14GB to 3.5GB and runs on ordinary GPUs; inference speeds up 2-3x; energy consumption drops. The cost is precision loss.

The industry's way of evaluating this cost has barely changed in decades:

• Perplexity (PPL): how "confused" the model is by a piece of text; lower is better. Measures language modeling ability.
• Downstream task accuracy**: how many questions the model answers correctly on benchmarks like HellaSwag, Winogrande, and ARC.

These two metrics are like coffee's "is it coffee" and "does it taste good." They tell you whether the model is still usable, but not how much it has changed in *how* it works.

The paper ran a very direct experiment. They quantized four models — Llama-3.2-3B, Vicuna-7B, Mistral-7B, and Llama-3.1-8B — using llama.cpp legacy quantization (Q8_0, Q5_0, Q4_0) and K-quantization (Q6_K down to Q2_K), compressing from 8 bits all the way to 2.

The perplexity results: Llama-3.2-3B's baseline PPL was 2.300, Q8_0 was 2.303, Q5_0 actually *dropped* to 2.170, and Q3_K fell to 1.967. Perplexity didn't rise — it fell. It looks like quantization made the model "less confused"?

But accuracy: baseline 55.5%, Q8_0 53.4%, Q3_K 51.5%. It is indeed declining, but mildly — seemingly no big deal.

This is where the "illusion of equivalency" comes from: the surface metrics are so stable that you assume nothing has changed.

Correctness Agreement: A Simple but Sharp New Metric

The paper's real contribution is a metric called Correctness Agreement (CA). Its definition is surprisingly simple:

> For each question, the base model got it right AND the quantized model also got it right — what fraction of all questions does this represent?

The formula is:

\[\text{CA}(c; \theta, \mathcal{D}) = \frac{1}{M} \sum_{m=1}^{M} \mathbb{1}[z_m = 1 \land z_m^{(c)} = 1]\]

where \(z_m\) is the base model's correct/incorrect label (1 or 0) on question \(m\), and \(z_m^{(c)}\) is the quantized model's label.

This metric has a clever property: it is always less than or equal to the *minimum* of the two models' accuracies. In other words, even if the quantized model's accuracy equals the base model's, CA can be far lower — because they may be answering *different* questions correctly.

This is the crux: two models can have identical accuracy while answering an entirely different set of questions correctly.

The experimental data is unsettling. Taking Llama-3.2-3B as an example:

| Quantization | Accuracy | CA | |---------|--------|-----| | Base | 55.5% | — | | Q8_0 | 53.4% | 41.4% | | Q6_K | 52.9% | 41.1% | | Q5_0 | 52.6% | 41.0% | | Q4_K | 52.1% | 40.9% | | Q3_K | 51.5% | 39.9% |

Look at accuracy: Q8_0 is only 2.1 points below baseline — seemingly acceptable. But look at CA: only 41.4%. This means that of the 55.5% of questions the base model answered correctly, roughly 14 points (about a quarter of the correct answers) became wrong after quantization. Meanwhile, the quantized model also correctly answered some questions the base model got wrong, so accuracy looks barely changed.

The model didn't "get a bit worse" — it "swapped the set of questions it answers correctly."

This finding has direct implications for practice. Imagine using an LLM for medical diagnostic assistance: the base model answers 55 of 100 cases correctly; after quantization, 53. Looks like a difference of only 2. But if 14 of those 53 differ from the base model's — that is, the quantized model gives different-but-correct diagnoses on 14 cases while failing on another 14 the base model got right — would you dare deploy it?

Quantization Is Not a Uniform Blade: Q/K Are More Fragile Than V/O

The paper's second major finding comes from structural analysis of attention weights.

A Transformer's self-attention layer has four projection matrices: Query (Q), Key (K), Value (V), and Output (O). Q determines "what I'm looking for," K determines "what I can offer," V carries the actual information content, and O is the output transformation.

The researchers computed statistical changes (mean, standard deviation, skewness, kurtosis) and distributional divergence (cosine similarity, Euclidean distance, KS statistic, KL divergence) for all four matrices after quantization. The results show a clear pattern:

Q and K projections are most sensitive to quantization; V and O are relatively stable.

Specifically, at Q3_K and Q2_K (3-bit and 2-bit quantization):

• Q and K show violent skewness fluctuations, collapsing kurtosis, and clear mean shifts
• V retains its low-kurtosis characteristics, with structure relatively intact
• O, though with the highest kurtosis, only degrades noticeably at the lowest bit widths

The KL divergence data is even more intuitive: as bit width decreases, Q and K weight distributions drift ever further from the base model, while V and O change far more mildly.

There is a natural analogy here. Imagine the attention mechanism as a large ball:

• Q is each guest's "person-finding radar" — "I'm looking for someone who knows machine learning"
• K is the identity tag on everyone's chest — "I work on NLP"
• V is what guests can actually talk about
• O is the social notes compiled after the party

Quantization is like dimming the lights. Q and K depend on fine-grained matching — the radar must distinguish "knows machine learning" from "knows machine repair," and the tags must separate "NLP" from "NLP papers." When the lights dim, these fine matchings fail first. V and O are more like recording what has already happened, less sensitive to lighting.

The paper thus offers an engineering suggestion: future quantization methods should allocate precision adaptively — more bits for Q and K, fewer for V and O. This is more sensible than one-size-fits-all uniform quantization.

Three Quantization Regimes: Safe, Degraded, Collapsed

By scanning the full range from 8-bit to 2-bit, the paper identifies three clear quantization regimes:

Safe zone (Q8_0 to Q4_K): statistical metrics barely change, perplexity is stable — but CA is already below the accuracy ceiling. Behavioral drift already exists; it's just masked by aggregate metrics.

Degraded zone (Q3_K): skewness starts to spike, kurtosis starts to collapse, KL divergence rises sharply. CA drops further. This is the turning point where structural damage begins to exceed functional compensation.

Collapse zone (Q2_K): all metrics deteriorate across the board. The weight distribution is unrecognizable compared to the base model. The model is no longer functionally trustworthy.

This trichotomy is very practical for practitioners. Today people choose quantization schemes mainly by "how much accuracy loss is acceptable." But the paper's data shows Q4_K's accuracy may be only 0.5 points below Q5_K's, while CA may differ by several points. You think you've lost 0.5% accuracy; in reality you've lost a quarter of the model's "decision identity."

Perplexity Can Lie

The paper has another finding worth singling out: there is a disconnect between perplexity and behavioral consistency.

Llama-3.2-3B's perplexity at Q3_K is 1.967, *lower* than the baseline's 2.300. By traditional standards, this is a "better" model. But its CA is only 39.9%, among the lowest of all schemes.

What does this mean? Perplexity measures the model's probability estimates over text; low PPL means the model is "not surprised" by the test text. But this lack of surprise may come from the smoothing effect introduced by quantization — extreme probabilities are compressed, all predictions become more "mediocre," and overall PPL falls. Yet the model's judgment on specific questions has already changed.

A model that is "never surprised" by anything is not necessarily a model with accurate judgment. It may simply have lost its sensitivity to nuance.

This recalls a psychology experiment: subjects are shown a series of face photos and asked to judge "is this person trustworthy." If you give the subjects blurry glasses, they may say "they're all fine" — the "variance" of overall judgments drops, but judgments about each individual face are already unreliable. Low perplexity is like this "blurry glasses effect."

Engineering Insights: What This Means

The paper offers several direct practical implications for AI practitioners:

1. Evaluating quantized models cannot rely on aggregate metrics alone. Accuracy and perplexity are necessary but insufficient. If your application is sensitive to *which* questions are answered correctly (medical, legal, safety moderation), you should add CA or similar behavioral consistency metrics.

2. Quantization scheme choice should follow the CA inflection point, not the accuracy inflection point. The paper's data shows Q4_K is the safe upper bound and Q3_K begins to degrade. This conclusion is more reliable than "stop when accuracy drops 2%."

3. Adaptive quantization has a clear direction. Q and K need higher precision; V and O can be compressed more aggressively. This provides an empirical basis for future mixed-precision quantization.

4. Low perplexity does not mean a good model. If you see PPL drop after quantization, don't celebrate too early — check behavioral consistency.

5. Quantization is never the "lossless compression" illusion. Even at 8-bit, CA is already clearly below the accuracy ceiling. Every quantization changes the model's decision identity — it's only a matter of degree.

A Deeper Thought: Identity and Capability

This paper raises a philosophical question: what is a model's "identity"?

The traditional view holds that a model's identity is its capability — how accurate it is, what it knows. As long as these don't change, it's still the same model.

But this paper offers another perspective: a model's identity lies in its judgments on specific cases. Two models with identical accuracy, answering different sets of questions correctly, are two different "decision-makers."

This matches the experience of human experts. Two doctors may both have 90% diagnostic accuracy, but if their judgments on specific cases frequently diverge, you'd say they are "doctors of different schools," not "equivalent doctors." A quantized model is essentially "a doctor who switched schools" — its overall level may be similar, but its judgment on specific cases has changed.

For AI systems that demand reproducibility, auditability, and accountability, this "identity drift" is a seriously underestimated risk. You deploy a quantized version based on the base model's test reports, assuming behavioral consistency. But on 14% of cases, they will give different judgments. Some of those differences may not surface on test sets, yet suddenly appear in the long-tail distribution of the real world.

The cost of quantization is not "getting a bit dumber" — it's "becoming another decision-maker." That other decision-maker may be just as smart, but it's smart in a different way.

Paper Information

• Title: The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs
• Authors: Baha Rababah, Cuneyt Gurcan Akcora, Carson K. Leung
• Institutions: University of Manitoba, Red River College, University of Central Florida
• arXiv: 2607.08734
• Quantization framework: llama.cpp
• Models tested: Llama-3.2-3B, Vicuna-7B, Mistral-7B, Llama-3.1-8B
• Benchmarks: WikiText-2, C4, HellaSwag, Winogrande, ARC

*Quantization is an engineering reality, not a choice we can avoid. But this paper reminds us: every precision compression changes the model's decision-making soul. Next time you see a report claiming "accuracy dropped only 1% after quantization," remember to ask: what about CA?*