Summary
A 2025 arXiv paper (2507.08705) by Baha Rababah, Cuneyt Gurcan Akcora, and Carson K. Leung challenges the standard evaluation of post-training quantization in large language models. The authors argue that accuracy and perplexity fail to capture behavioral changes induced by quantization. They introduce correctness agreement, a decision-level metric measuring the overlap in correct predictions between a base model and its quantized variants, independent of absolute accuracy. Across multiple models and quantization schemes from 8-bit to 2-bit, behavioral divergence emerges under moderate quantization even when task performance appears preserved. Analyzing quantization as a structural operation on attention weights using statistical and distributional measures, the study reveals non-linear breakpoints at low bit-widths and shows that query and key projections are consistently more sensitive to quantization than value and output projections.
Paper Overview
Research Area: LLM Quantization
Authors: Baha Rababah, Cuneyt Gurcan Akcora, Carson K. Leung
Published: 2025-07-12
arXiv: 2507.08705
Summary
Post-training quantization is widely used to deploy large language models in resource-constrained settings, yet its evaluation relies almost exclusively on accuracy and perplexity. The authors show that these metrics fail to capture behavioral changes induced by quantization.
They introduce correctness agreement, a decision-level metric that measures the overlap in correct predictions between a base model and its quantized variants, independent of absolute accuracy.
Key Findings
- Across multiple models and quantization schemes from 8-bit to 2-bit, behavioral divergence emerges under moderate quantization even when task performance appears preserved.
- The authors analyze quantization as a structural operation on attention weights, using statistical and distributional measures to quantify layer-wise distortion.
- Results reveal non-linear breakpoints at low bit-widths.
- Query and key projections are consistently more sensitive to quantization than value and output projections.
Links
- Paper: https://arxiv.org/abs/2507.08705
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178379431