The Illusion of Equivalency: Quantization Changes LLM Behavior Even When Accuracy Looks Fine

Forum topic · 小凯 · 2026-07-13

Summary

A 2025 arXiv paper (2507.08705) by Baha Rababah, Cuneyt Gurcan Akcora, and Carson K. Leung argues that standard evaluation metrics for post-training quantization of large language models—accuracy and perplexity—fail to capture quantization-induced behavioral changes. The authors introduce correctness agreement, a decision-level metric measuring the overlap in correct predictions between a base model and its quantized variants, independent of absolute accuracy. Experiments across multiple models and quantization schemes from 8-bit to 2-bit show behavioral divergence emerging under moderate quantization even when task performance appears preserved. To explain this, the authors treat quantization as a structural operator on attention weights and quantify layer-wise distortion using statistical and distributional measures, revealing nonlinear breakpoints at low bit-widths. Query and key projections are consistently more sensitive to quantization than value and output projections. The findings expose an illusion of equivalency between base and quantized models and motivate behavioral evaluation beyond traditional performance metrics.

Overview

Field: Machine Learning Authors: Baha Rababah, Cuneyt Gurcan Akcora, Carson K. Leung Published: 2025-07-12 arXiv: 2507.08705

Abstract

Post-training quantization is widely used to deploy large language models in resource-constrained settings, yet its evaluation relies almost exclusively on accuracy and perplexity. This paper shows that these metrics fail to capture behavioral changes induced by quantization.

The authors introduce correctness agreement, a decision-level metric that measures overlap in correct predictions between a base model and its quantized variants, independent of absolute accuracy.

Key findings

Across multiple models and quantization schemes from 8-bit to 2-bit, behavioral divergence emerges under moderate quantization even when task performance appears preserved.
Quantization can be analyzed as a structural operator on attention weights, with layer-wise distortion quantified using statistical and distributional measures.
Results reveal nonlinear breakpoints at low bit-widths.
Query and key projections are consistently more sensitive to quantization than value and output projections.

These findings expose an illusion of equivalency between base and quantized models and argue for behavioral evaluation beyond traditional performance metrics.