The Illusion of Equivalency: Statistical Characterization of Quantization Damage in LLM Attention Weights

Forum topic · 小凯 · 2026-07-13

Summary

A 2025 arXiv paper (2507.08705) by Baha Rababah, Cuneyt Gurcan Akcora, and Carson K. Leung argues that post-training quantization of large language models is inadequately evaluated by accuracy and perplexity alone. The authors introduce correctness agreement, a decision-level metric measuring the overlap in correct predictions between a base model and its quantized variant, independent of absolute accuracy. Across multiple models and quantization schemes from 8-bit to 2-bit, they find behavioral divergence emerges under moderate quantization even when task performance appears preserved. To explain this, they analyze quantization as a structural operator on attention weights and quantify layer-wise distortions using statistical and distributional measures. Results reveal nonlinear breakpoints at low bit-widths and show that query and key projections are consistently more sensitive to quantization than value and output projections. The findings highlight an illusion of equivalency between base and quantized models and motivate behavioral evaluation beyond conventional performance metrics.

Paper Overview

Field: Machine Learning
Authors: Baha Rababah, Cuneyt Gurcan Akcora, Carson K. Leung
Published: 2025-07-12
arXiv: 2507.08705

Abstract

Post-training quantization is widely used to deploy large language models in resource-constrained settings, yet its evaluation relies almost exclusively on accuracy and perplexity. The authors show that these metrics fail to capture behavioral changes induced by quantization.

They introduce correctness agreement, a decision-level metric that measures overlap in correct predictions between a base model and its quantized variant, independent of absolute accuracy. Across multiple models and quantization schemes from 8-bit to 2-bit, they find that behavioral divergence emerges under moderate quantization even when task performance appears preserved.

To explain this effect, the paper analyzes quantization as a structural operator on attention weights and quantifies layer-wise distortions using statistical and distributional measures. The results reveal nonlinear breakpoints at low bit-widths and show that query and key projections are consistently more sensitive to quantization than value and output projections.

Key Findings

Accuracy and perplexity alone do not capture quantization-induced behavioral changes.
Behavioral divergence appears under moderate quantization even when task performance seems preserved — an "illusion of equivalency."
Query and key projections are more quantization-sensitive than value and output projections.
The authors advocate behavioral evaluation beyond traditional performance metrics.

--- *Auto-collected on 2026-07-13*