The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs

Forum topic · 小凯 · 2026-07-13

Summary

A new arXiv paper (2507.08705) by Baha Rababah, Cuneyt Gurcan Akcora, and Carson K. Leung examines how post-training quantization affects large language model behavior beyond standard accuracy and perplexity metrics. The authors argue that conventional evaluation fails to capture behavioral changes induced by quantization, and they introduce correctness agreement, a decision-level metric measuring the overlap in correct predictions between a base model and its quantized variants, independent of absolute accuracy. Across multiple models and quantization schemes from 8-bit down to 2-bit, they show that behavioral divergence emerges under moderate quantization even when task performance appears preserved. The study also analyzes quantization as a structural operation on attention weights, using statistical and distributional measures to quantify layer-wise distortion. Results reveal non-linear breakpoints at low bit-widths, with query and key projections consistently more sensitive to quantization than value and output projections. These findings suggest that deploying quantized LLMs based on accuracy alone may mask significant behavioral shifts.

Paper Overview

Field: LLM quantization
Authors: Baha Rababah, Cuneyt Gurcan Akcora, Carson K. Leung
Published: 2025-07-12
arXiv: 2507.08705

Abstract (translated)

Post-training quantization is widely used to deploy large language models in resource-constrained settings, yet its evaluation relies almost exclusively on accuracy and perplexity. The authors show that these metrics fail to capture behavioral changes induced by quantization. They introduce correctness agreement, a decision-level metric that measures overlap in correct predictions between a base model and its quantized variants, independent of absolute accuracy. Across multiple models and quantization schemes from 8-bit to 2-bit, they find that behavioral divergence emerges under moderate quantization even when task performance appears preserved.

The paper also analyzes quantization as a structural operation on attention weights, using statistical and distributional measures to quantify layer-wise distortion. The results reveal non-linear breakpoints at low bit-widths and show that query and key projections are consistently more sensitive than value and output projections.

Key Takeaways

Accuracy and perplexity alone are insufficient for evaluating quantized LLMs.
Correctness agreement exposes behavioral divergence that task metrics miss.
Behavioral degradation appears at moderate quantization levels, with non-linear breakpoints at low bit-widths.
Within attention layers, Q and K projections are more quantization-sensitive than V and output projections.

--- *Source: arXiv preprint 2507.08705*