Hidden Pitfalls of Quantized LLM Deployments: A Deep Dive into Baidu Research's Maximum Activation Measurement Study
> When you quantize a large model to INT8 for deployment, you might assume that the maximum activation value grows monotonically with parameter scale. But a recent Baidu Research study shows this assumption is entirely wrong. In a unified measurement across 8 model families and 27 checkpoints, maximum activations span nearly 4 orders of magnitude — from 122 (Qwen3.5) to 696,320 (Gemma3-27B-it). Models with comparable parameter counts can differ by several orders of magnitude in activation peaks. What does this mean? Your quantization scheme may work perfectly on some models while failing badly on others, where a few extreme activations distort the entire quantization batch. This paper is the first to systematically transform "massive activations" from a binary outlier criterion into a continuous, deployment-relevant statistic, uncovering key patterns including MoE's hidden advantage, SFT's layer-selective compression, and the monotonic amplification effect of training progress. The measurement pipeline is open-sourced on GitHub.
---
1. The Problem: Why Quantized Deployment Involves Guesswork
Low-bit quantization is a necessary step for on-device LLM deployment. For INT8, INT4, and lower-precision schemes, one of the core problems is choosing the activation scale — a scaling factor that covers the dynamic range of activation values.
The traditional approach: run a forward pass over a calibration dataset, find the maximum activation value, and pick a scale that covers it.
But this approach rests on two hidden assumptions: 1. Maximum activations correlate positively with parameter scale — larger models have larger activations 2. Different model families have maximum activations in the same order of magnitude — similar schemes can be applied
This Baidu Research paper proves that both assumptions are wrong.
---
2. Paper Overview: One-Sentence Positioning
> "We measured global maximum activations for 8 post-LLaMA-era open-source model families and 27 checkpoints under a unified protocol, finding they span nearly 4 orders of magnitude with no monotonic relationship to parameter scale. Maximum activation is a property tied to family, architecture, and training stage — not a simple byproduct of scale."
Paper information:
- Title: *Measuring Maximum Activations in Open Large Language Models*
- Authors: Luxuan Chen, Han Tian, Xinran Chen, Rui Kong, Fang Wang, Jiamin Chen, Yuchen Li, Jiashu Zhao, Shuaiqiang Wang, Haoyi Xiong, Linghe Kong, Dawei Yin
- Institutions: Shanghai Jiao Tong University, Baidu, Nankai University
- arXiv: 2605.15572
- Code: https://github.com/clx1415926/Max_act_llm
- Core innovation: transforming "massive activations" from a binary criterion into a continuous, deployment-relevant statistic
- 256 tokens: 1%
- 512 tokens: 1%
- 1024 tokens: 2%
- 2048 tokens: 3%
- 4096 tokens: 93% (aligned with real long-context deployment)
- Average length: ~3,899 tokens
- Total tokens: ~19.5M
- Max coefficient of variation with 1,000 samples: 10.1% (Qwen3-30B-A3B)
- Max coefficient of variation with 2,000 samples: 8.2%
- At comparable parameter counts (27B–32B), global maximum activation ranges from 132 (Qwen3.5-35B-A3B) to 696,320 (Gemma3-27B-it) — a 5,260x difference
- These are not "outliers" — they are systematic differences determined by family-level architectural and training choices
- Quantization schemes that only examine attention or MLP outputs will miss the true peaks
- Activation scaling strategies should prioritize peak detection in hidden states
- MoE's sparse routing naturally distributes activation load
- Averaging over expert activations smooths extremes
- This means MoE models may suit low-bit quantization better than dense models
- Mid/upper-layer peak regions: 1.000x (nearly unchanged after SFT)
- Final-layer peak: 1,536 → 840 (-45%, 1.5B)
- Final-layer peak: 4,864 → 2,528 (-48%, 7B)
- Final-layer peak: 30,848 → 21,248 (-31%, 32B)
- Condition 1: \|x_i\| > 100
- Condition 2: \|x_i\| / median(\|x_j\|) > 1000
- Qwen3.5-0.8B (global max 122) fails the Sun criterion but is easiest to quantize
- Gemma3-27B-it (global max 696,320) passes the criterion and is hardest to quantize
- But Qwen2.5-1.5B (global max 7,968) also passes the criterion, with moderate quantization difficulty
- Larger global maximum M → activation scale is chosen to fit the extreme → lower quantization resolution for typical activations → significantly lower INT8 signal-to-quantization-noise ratio (SQNR)
- This directly demonstrates the practical deployment value of measuring maximum activations
- Qwen2.5 → Qwen3: activation peaks rise
- Qwen3 → Qwen3.5: activation peaks drop sharply (strongly suppressed)
- Gemma2 → Gemma3: activation peaks increase substantially
- 3B, 7B, 32B: text-only vs multimodal — significant differences at each size
- Global maximum activation M = max\|a\|
- Layer-wise peak locations
- Peak-carrying component (residual stream / attention / MLP)
- Qwen3.5-35B-A3B (35B total parameters) has a peak of 132
- Qwen2.5-1.5B (1.5B parameters) has a peak of 7,968
- MoE models may achieve better SQNR under INT8 and even INT4 quantization
- For on-device deployment, MoE's quantization robustness may be an underrated advantage
- Prioritize peak detection on hidden states
- Rather than only inspecting attention or MLP outputs
- Paper: *Measuring Maximum Activations in Open Large Language Models*
- Authors: Luxuan Chen, Han Tian, Xinran Chen, Rui Kong, Fang Wang, Jiamin Chen, Yuchen Li, Jiashu Zhao, Shuaiqiang Wang, Haoyi Xiong, Linghe Kong, Dawei Yin
- Institutions: Shanghai Jiao Tong University, Baidu, Nankai University
- Corresponding authors: Yuchen Li (yuchenli1230@gmail.com), Dawei Yin (yindawei@acm.org)
- arXiv: 2605.15572
- Code: https://github.com/clx1415926/Max_act_llm
- Measurement scale: 27 checkpoints, 8 model families, 5,000-sample multi-domain corpus, ~19.5M tokens
- Components covered: embeddings, hidden states, attention outputs, MLP/MoE outputs, SwiGLU gates, final normalization
- Key results: global max activation 122–696,320 (nearly 4 orders of magnitude); MoE peak reduction 14–23.4x; residual stream carries 22/24 global maxima; training progress monotonically increases peaks 1.34x; SFT late-layer compression 1.4x
- Quantization validation: INT8 SQNR co-varies with maximum activation
- Stability: coefficient of variation < 10.1%
---
3. Unified Measurement Protocol: 5,000 Samples, 6 Component Types, 8 Families
3.1 Corpus Design (5,000 samples, multi-domain)
| Domain | Samples | Share | Notes | |---|---|---|---| | Math/Science | 850 | 17% | Formula-dense | | Code | 850 | 17% | Code snippets | | English web | 850 | 17% | Natural language | | Knowledge/QA/Books | 850 | 17% | Encyclopedic, Q&A | | Chinese | 400 | 8% | Multilingual coverage | | Other low-resource languages | 300 | 6% | Diversity coverage | | Additional English/mixed web | 900 | 18% | Increased web share | | Total | 5,000 | 100% | |
Length control:
Key design choice: text content is fixed, with separate tokenization per model family, avoiding activation-statistics bias from different tokenizers.
3.2 Model Suite (27 checkpoints, 8 families)
| Family | Checkpoints | Role | |---|---|---| | Qwen2.5 | 1.5B, 7B, 32B | Dense scaling baseline | | Qwen2.5-VL | 3B, 7B, 32B | Vision-language adaptation | | Qwen3 | 1.7B, 8B, 30B-A3B, 32B | Dense/MoE comparison | | Qwen3.5 | 0.8B, 9B, 27B, 35B-A3B | Low-outlier counterexample family | | Gemma2 | 2B, 9B, 27B | Non-monotonic scaling case | | Gemma3 | 4B-it, 27B-it | High-magnitude family | | Ling | 5T, 10T, 15T, 20T | Training-stage evolution | | GPT-OSS | 20B | Component-level special case | | Qwen2.5-Instruct | 1.5B, 7B, 32B | SFT comparison (additional) |
3.3 Six Types of Activation Tensors Collected
Full-layer capture via PyTorch forward hooks: 1. Embedding outputs 2. Per-layer hidden states (layer outputs after residual updates) 3. Attention outputs 4. MLP/MoE outputs 5. SwiGLU gate pre-activations (pre-activation values in gated MLPs) 6. Final LayerNorm outputs
3.4 Measurement Stability Verification
For 4 representative models, stratified subsamples of 1,000 and 2,000 samples were constructed and repeated 5 times:
Conclusion: the observed maximum activations are not incidental single-sample errors; the statistics are robust.
---
4. Core Findings: From Binary Outliers to Continuous Peaks
4.1 Finding 1: A Near 4-Order-of-Magnitude Cross-Family "Activation Gap"
Global maximum activation panorama:
| Rank | Model | Global max \|activation\| | Family | Notes | |---|---|---|---|---| | 1 | Gemma3-27B-it | 696,320 | Gemma3 | Highest publicly measured | | 2 | Gemma3-4B-it | 245,760 | Gemma3 | Very high even at small scale | | 3 | Gemma2-27B | 157,696 | Gemma2 | Non-monotonic: 9B < 2B | | 4 | GPT-OSS-20B | 43,008 | GPT-OSS | MLP-output exception | | 5 | Qwen3-32B | 35,328 | Qwen3 | Dense, high value | | 6 | Qwen2.5-32B | 30,848 | Qwen2.5 | In-family scaling | | Low | Qwen3.5-0.8B | 122 | Qwen3.5 | Lowest | | Low | Qwen3.5-35B-A3B | 132 | Qwen3.5 | MoE, low value | | Low | Qwen3.5-27B | 167 | Qwen3.5 | Low value | | Low | Qwen3-30B-A3B | 1,512 | Qwen3 | MoE, low value |
Key insights:
4.2 Finding 2: The Residual Stream Is the Primary Carrier of Extremes
In 22 of 24 main-analysis checkpoints, the global maximum appears in the residual stream (hidden states).
Implications:
Exception: GPT-OSS-20B carries its global maximum in MLP outputs, showing different architectures need different inspection strategies.
4.3 Finding 3: MoE's Hidden Advantage — Peaks 14–23.4x Lower
| Comparison | Dense peak | MoE peak | Reduction | |---|---|---|---| | Qwen3-8B vs 30B-A3B | 17,664 | 1,512 | 11.7x | | Qwen3.5-9B vs 35B-A3B | 956 | 132 | 7.2x | | (other comparisons in the paper) | | | 14.0–23.4x |
Why are MoE activation peaks lower?
4.4 Finding 4: SFT Mainly Compresses Late Layers — Mid-Layer Peaks Unchanged
Qwen2.5 Base vs Instruct comparison:
| Scale | Base peak | Instruct peak | Change | Main effect | |---|---|---|---|---| | 1.5B | 7,968 | 7,968 | 1.000x | None | | 7B | 13,248 | 13,312 | 1.005x | Almost none | | 32B | 30,848 | 22,144 | 1.4x↓ | Late-layer compression |
Layer-level details:
Conclusion: SFT mainly compresses late-layer peaks, exposing pre-existing high-peak regions in middle layers as the new global maximum. It doesn't "eliminate" outliers — it "moves" the global maximum's location.
4.5 Finding 5: Training Progress Monotonically Amplifies Activation Peaks
Ling-mini series (fixed architecture and scale, different training token counts):
| Training stage | Global max activation | Relative to 5T | |---|---|---| | 5T | 7,648 | 1.00x | | 10T | 9,024 | 1.18x | | 15T | 9,600 | 1.26x | | 20T | 10,240 | 1.34x |
Conclusion: even with family, architecture, and parameter count fixed, training progress itself monotonically increases extreme activation magnitude. Training stage should be recorded as an independent observational dimension, alongside model size and architecture.
4.6 Finding 6: Binary Criteria Mismatch Actual Quantization Difficulty
Sun et al.'s binary "massive activation" criterion:
The paper finds this criterion inconsistent with deployment quantization difficulty:
Core problem: a binary test only asks "does it exist," not "how large is it." Deployment quantization cares about actual magnitude — 696,320 and 7,968 imply completely different scale choices.
4.7 Finding 7: INT8 Signal-to-Noise Ratio Co-varies with Maximum Activation
A lightweight INT-8 sanity check validates this:
---
5. Five Matched Designs: Controlled-Variable Decomposition
5.1 Scaling Within Family
| Family | 1.5B–3B | 7B–9B | 27B–32B | Trend | |---|---|---|---|---| | Qwen2.5 | 7,968 | 13,248 | 22,144 | Monotonic increase | | Qwen2.5-VL | 3,248 | 8,256 | 22,144 | Monotonic increase | | Qwen3 | 14,208 | 17,664 | 35,328 | Monotonic increase | | Qwen3.5 | 122 | 956 | 167 | Non-monotonic! | | Gemma2 | 2,992 | 1,656 | 157,696 | Non-monotonic! | | Gemma3 | — | — | 245,760 / 696,320 | Monotonic (only two points) |
Key observation: Gemma2's 9B peak (1,656) is below its 2B peak (2,992), then 27B surges to 157,696. It's not simply "bigger = more extreme."
5.2 Cross-Generation Evolution
The Qwen family shows an inverted-V trajectory:
The Gemma family rises in a straight line:
Conclusion: maximum activation neither monotonically shrinks nor grows with release date — it is highly family-dependent.
5.3 Vision-Language vs Text-Only
Qwen2.5 vs Qwen2.5-VL at matched scales:
Vision-language adaptation has a measurable effect on activation magnitude, but smaller than cross-family differences.
---
6. Five Practical Guidelines for Deployment
6.1 Publish Maximum Activation as a Standard Model-Card Attribute
Current open-weight model cards typically include parameter scale, architecture, training data, and license. The paper suggests adding:
This lets deployers gauge quantization difficulty before low-bit deployment.
6.2 Family-Level Differences Exceed Scale Differences
Don't assume "a 32B model is harder to quantize than a 7B model." In fact:
A 35B MoE model may be easier to quantize than a 1.5B dense model.
6.3 The Quantization Advantage of MoE Architectures
MoE checkpoints have activation peaks 14–23.4x lower than matched dense models. This means:
6.4 The Residual Stream Is the First Inspection Target
22/24 checkpoints carry their global maximum in the residual stream. Quantization tools should:
6.5 The Necessity of Recording Training Stage
Ling-mini data shows peaks increased 1.34x from 5T to 20T training tokens. If an open model releases a mid-training checkpoint (e.g., 5T), deployers should know its peaks may be lower than the final version's — and lower peaks here do NOT mean the final version is easier to quantize; quite the opposite, the final version may be harder.
---
7. Limitations and Future Directions
1. Correlation, not causation: the paper identifies associations without mechanisms. Why are Gemma3's peaks so high — normalization choices, initialization, training data, or something else? Further research needed. 2. Incomplete family coverage: Mistral, Llama 3, DeepSeek, and others are not included. 3. No dynamic inference-time activations: measurements use an offline calibration set; real user inputs may trigger larger or smaller activations. 4. No interventions: the paper only measures. How to reduce peaks via post-processing or architectural changes is future work.
---
8. One-Sentence Summary
The paper's core contribution is reframing "activation outliers": from a binary judgment (do outliers exist?) to a continuous statistic (how large is the actual maximum?), and proving that this number is a model property rather than a byproduct of scale. When Gemma3-27B-it's peak is 5,260x that of Qwen3.5-35B-A3B, any intuition based on "bigger models are harder to quantize" will mislead you. MoE's 14–23.4x peak reduction, SFT's late-layer compression, and training's monotonic amplification are not paper footnotes — they are hidden traps deployment engineers face every day. Baidu Research's recommendation is direct: when releasing open weights, put the maximum activation M and its layer-wise location in the model card. That's not academic advice; it's a survival guide for production environments.
---
References