The Hidden Pitfall of Open LLM Quantization: A Deep Dive into Baidu Research's Maximum Activation Study

> When you quantize a large model to INT8 for deployment, you may assume maximum activation values grow monotonically with parameter scale. A new study from Baidu Research shows this assumption is completely wrong. Across a unified measurement of 8 model families and 27 checkpoints, maximum activations span nearly 4 orders of magnitude—from 122 for Qwen3.5 to 696,320 for Gemma3-27B-it. At equal parameter counts, activation peaks can differ between families by several orders of magnitude. Your quantization recipe may work perfectly on some models and distort entire quantization batches on others due to a few extreme activations. This work is the first to systematically transform "maximum activation" from a binary outlier criterion into a continuous, deployment-relevant statistic, uncovering key regularities such as MoE's hidden advantage, SFT's layer-selective compression, and training's monotonic amplification effect. The measurement pipeline is open-sourced on GitHub.

1. The Problem: Quantization Deployment Relies on Guesswork

Low-bit quantization is essential for on-device LLM deployment. For INT8, INT4, and lower precisions, a core problem is choosing the activation scale—the scaling factor covering the dynamic range of activations.

The traditional approach: run a forward pass on a calibration set, find the maximum activation, and pick a scale that covers it.

But this approach carries two implicit assumptions: 1. Maximum activations correlate positively with parameter scale — bigger models have bigger activations. 2. Different model families have maximum activations in the same order of magnitude — similar recipes can be applied.

This paper proves both assumptions are wrong.

2. Paper Overview

> "We measure, for the first time under a unified protocol, the global maximum activations of 8 post-LLaMA-era open model families across 27 checkpoints, finding they span nearly 4 orders of magnitude with no monotonic relationship to scale. Maximum activation is a family-, architecture-, and training-stage-dependent model property, not a simple byproduct of scale."

Paper info:

• Title: Measuring Maximum Activations in Open Large Language Models
• Authors: Luxuan Chen, Han Tian, Xinran Chen, Rui Kong, Fang Wang, Jiamin Chen, Yuchen Li, Jiashu Zhao, Shuaiqiang Wang, Haoyi Xiong, Linghe Kong, Dawei Yin
• Institutions: Shanghai Jiao Tong University, Baidu, Nankai University
• arXiv: 2605.15572
• Code: https://github.com/clx1415926/Max_act_llm
• Core contribution: reframing "massive activations" from a binary criterion into a continuous, deployment-relevant statistic

3. Unified Measurement Protocol: 5,000 Samples, 6 Component Types, 8 Families

3.1 Corpus Design (5,000 samples, multi-domain)

| Domain | Samples | Share | Notes | |--------|---------|-------|-------| | Math/science | 850 | 17% | Formula-heavy | | Code | 850 | 17% | Code snippets | | English web | 850 | 17% | Natural language | | Knowledge/QA/books | 850 | 17% | Encyclopedic, Q&A | | Chinese | 400 | 8% | Multilingual coverage | | Other low-resource languages | 300 | 6% | Diversity | | Extra English/mixed web | 900 | 18% | Higher web share | | Total | 5,000 | 100% | |

Length control: 1% at 256 tokens, 1% at 512, 2% at 1024, 3% at 2048, and 93% at 4096 tokens (matching real deployment's long-context use). Average length ~3,899 tokens; total ~19.5M tokens.

Key design: fixed text content, tokenized separately per model family to avoid tokenizer-induced bias in activation statistics.

3.2 Model Suite (27 checkpoints, 8 families)

| Family | Checkpoints | Role | |--------|-------------|------| | Qwen2.5 | 1.5B, 7B, 32B | Dense scaling baseline | | Qwen2.5-VL | 3B, 7B, 32B | Vision-language adaptation | | Qwen3 | 1.7B, 8B, 30B-A3B, 32B | Dense/MoE comparison | | Qwen3.5 | 0.8B, 9B, 27B, 35B-A3B | Low-outlier counterexample family | | Gemma2 | 2B, 9B, 27B | Non-monotonic scaling case | | Gemma3 | 4B-it, 27B-it | High-magnitude family | | Ling | 5T, 10T, 15T, 20T | Training-stage evolution | | GPT-OSS | 20B | Component-level special case | | Qwen2.5-Instruct | 1.5B, 7B, 32B | SFT comparison (additional) |

3.3 Six Activation Tensor Types Collected

Using PyTorch forward hooks across all layers: 1. Embedding outputs 2. Per-layer hidden states (post-residual-update layer outputs) 3. Attention outputs 4. MLP/MoE block outputs 5. SwiGLU gated pre-activations 6. Final LayerNorm outputs

3.4 Measurement Stability

For 4 representative models, stratified subsamples of 1,000 and 2,000 samples were repeated 5 times:

• Max coefficient of variation at 1,000 samples: 10.1% (Qwen3-30B-A3B)
• Max CV at 2,000 samples: 8.2%

Conclusion: observed maxima are not single-sample flukes; results are statistically robust.

4. Core Findings

4.1 Finding 1: A Near-4-Order-of-Magnitude Cross-Family "Activation Gap"

| Rank | Model | Global max |activation| | Family | Notes | |------|-------|-----------|--------|-------| | 1 | Gemma3-27B-it | 696,320 | Gemma3 | Highest public | | 2 | Gemma3-4B-it | 245,760 | Gemma3 | Very high even at small scale | | 3 | Gemma2-27B | 157,696 | Gemma2 | Non-monotonic: 9B < 2B | | 4 | GPT-OSS-20B | 43,008 | GPT-OSS | MLP-output exception | | 5 | Qwen3-32B | 35,328 | Qwen3 | Dense high value | | 6 | Qwen2.5-32B | 30,848 | Qwen2.5 | Within-family scaling | | ... | ... | ... | ... | ... | | Low | Qwen3.5-0.8B | 122 | Qwen3.5 | Lowest | | Low | Qwen3.5-35B-A3B | 132 | Qwen3.5 | MoE low value | | Low | Qwen3.5-27B | 167 | Qwen3.5 | Low value | | Low | Qwen3-30B-A3B | 1,512 | Qwen3 | MoE low value |

Key insight: at comparable parameter counts (27B–32B), global max activation ranges from 132 (Qwen3.5-35B-A3B) to 696,320 (Gemma3-27B-it)—a 5,260x difference. This is not an "outlier"—it is a systematic difference determined by family-level architecture and training choices.

4.2 Finding 2: The Residual Stream Is the Primary Carrier of Extreme Values

In 22 of 24 main-analysis checkpoints, the global maximum occurs in the residual stream (hidden states).

Implications:

• Quantization recipes that only inspect attention or MLP outputs will miss the true peaks
• Activation scaling strategies should prioritize peak detection in hidden states

Exception: GPT-OSS-20B's global maximum lies in MLP outputs—different architectures require different inspection strategies.

4.3 Finding 3: MoE's Hidden Advantage — Peaks Reduced 14–23.4x

| Pair | Dense model | Dense peak | MoE model | MoE peak | Reduction | |------|-------------|------------|-----------|----------|-----------| | Qwen3-8B vs 30B-A3B | Qwen3-8B | 17,664 | Qwen3-30B-A3B | 1,512 | 11.7x | | Qwen3.5-9B vs 35B-A3B | Qwen3.5-9B | 956 | Qwen3.5-35B-A3B | 132 | 7.2x | | (other paper comparisons) | | | | | 14.0–23.4x |

Why are MoE peaks lower?

• Sparse routing naturally distributes activation load
• Averaging over expert activations smooths extremes
• Implication: MoE models may suit low-bit quantization better than dense models

4.4 Finding 4: SFT Mostly Compresses Late Layers — Mid-Layer Peaks Unchanged

Qwen2.5 Base vs Instruct, layer-level comparison:

| Scale | Base peak | Instruct peak | Change | Main effect | |-------|-----------|---------------|--------|-------------| | 1.5B | 7,968 | 7,968 | 1.000x | No change | | 7B | 13,248 | 13,312 | 1.005x | Nearly none | | 32B | 30,848 | 22,144 | 1.4x down | Late-layer compression |

Layer details:

• Mid-to-high layer peak region: 1.000x (unchanged by SFT)
• Final layer peak: 1,536 → 840 (-45%, 1.5B)
• Final layer peak: 4,864 → 2,528 (-48%, 7B)
• Final layer peak: 30,848 → 21,248 (-31%, 32B)

Conclusion: SFT mainly compresses late-layer peaks, exposing existing high-peak regions in mid layers as the new global maximum. It doesn't "eliminate" outliers—it "moves" the location of the global maximum.

4.5 Finding 5: Training Progression Monotonically Amplifies Peaks

Ling-mini series (fixed architecture and scale, varying training tokens):

| Training stage | Global max activation | Relative to 5T | |----------------|----------------------|----------------| | 5T | 7,648 | 1.00x | | 10T | 9,024 | 1.18x | | 15T | 9,600 | 1.26x | | 20T | 10,240 | 1.34x |

Conclusion: even with fixed family, architecture, and parameters, training progression monotonically increases extreme activation magnitudes. Training stage should be recorded as an independent observation dimension alongside model size and architecture.

4.6 Finding 6: Binary Criteria Mismatch Actual Quantization Difficulty

Sun et al.'s binary "massive activation" criterion:

• Condition 1: |x_i| > 100
• Condition 2: |x_i| / median(|x_j|) > 1000

The paper finds this criterion is inconsistent with deployment-side quantization difficulty:

• Qwen3.5-0.8B (global max 122) fails the Sun criterion but is easiest to quantize
• Gemma3-27B-it (global max 696,320) passes the Sun criterion and is hardest to quantize
• But Qwen2.5-1.5B (global max 7,968) also passes the Sun criterion, with moderate quantization difficulty

Core problem: binary criteria only ask "does it exist?", not "how large is it?". Deployment cares about actual magnitude—696,320 and 7,968 demand completely different scale choices.

4.7 Finding 7: INT8 Signal-to-Quantization-Noise Ratio Co-varies with Maximum Activation

A lightweight INT8 sanity check confirms:

• Larger global max activation M → activation scale skewed toward extremes → reduced quantization resolution for ordinary activations → significantly lower INT8 reconstruction SQNR
• This directly demonstrates the practical deployment relevance of measuring maximum activations.

5. Five Matched Designs: Controlled-Variable Decomposition

5.1 Scaling Within Family

| Family | 1.5B–3B | 7B–9B | 27B–32B | Trend | |--------|---------|-------|---------|-------| | Qwen2.5 | 7,968 | 13,248 | 22,144 | Monotonic growth | | Qwen2.5-VL | 3,248 | 8,256 | 22,144 | Monotonic growth | | Qwen3 | 14,208 | 17,664 | 35,328 | Monotonic growth | | Qwen3.5 | 122 | 956 | 167 | Non-monotonic! | | Gemma2 | 2,992 | 1,656 | 157,696 | Non-monotonic! | | Gemma3 | — | — | 245,760 / 696,320 | Growth (only two points) |

Key observation: Gemma2's 9B peak (1,656) is lower than its 2B peak (2,992), then the 27B surges to 157,696. It is not simply "bigger = more extreme."

5.2 Cross-Generation Evolution

The Qwen family shows an inverted-V trajectory:

• Qwen2.5 → Qwen3: activation peaks rise
• Qwen3 → Qwen3.5: activation peaks fall sharply (strongly suppressed)

The Gemma family rises steadily:

• Gemma2 → Gemma3: activation peaks rise sharply

Conclusion: maximum activation neither monotonically shrinks nor grows with release date—it is highly family-dependent.

5.3 Vision-Language vs Text-Only

Same-scale Qwen2.5 vs Qwen2.5-VL comparisons show significant differences at 3B, 7B, and 32B. Vision-language adaptation measurably affects activation magnitude, but less than cross-family differences.

6. Five Deployment Takeaways

6.1 Publish Maximum Activation as a Standard Model Card Attribute

Current model cards typically list parameter scale, architecture, training data, and license. The paper suggests adding:

• Global maximum activation M = max|a|
• Layer-wise peak locations
• Peak-carrying component (residual stream / attention / MLP)

Deployers could then gauge quantization difficulty before low-bit deployment.

6.2 Family-Level Differences Exceed Scale Differences

Don't assume "a 32B model is harder to quantize than a 7B." In fact:

• Qwen3.5-35B-A3B (35B total parameters) has a peak of 132
• Qwen2.5-1.5B (1.5B parameters) has a peak of 7,968

A 35B MoE model may be easier to quantize than a 1.5B dense model.

6.3 MoE Quantization Advantage

MoE checkpoints show activation peaks 14–23.4x lower than scale-matched dense models, suggesting:

• Better SNR under INT8 or even INT4 quantization
• An underappreciated robustness advantage of MoE for on-device deployment

6.4 The Residual Stream Is the First Place to Check

22/24 checkpoints carry their global maximum in the residual stream. Quantization tools should:

• Prioritize peak detection on hidden states
• Not rely solely on attention or MLP outputs

6.5 The Need to Record Training Stage

Ling-mini data shows peaks grow 1.34x from 5T to 20T training tokens. If an open release corresponds to an intermediate stage (e.g., a 5T checkpoint), deployers should know the peak may be lower than the final version—and that the final version is likely *harder* to quantize, not easier.

7. Limitations and Future Directions

1. Correlation, not causation: the paper identifies associations without establishing mechanisms. Why are Gemma3's peaks so high—normalization choices, initialization, training data, or something else? Further work needed. 2. Incomplete family coverage: Mistral, Llama 3, DeepSeek, and others are not included. 3. No dynamic inference-time activations: measurements use an offline calibration set; real user inputs may trigger larger or smaller activations. 4. No intervention: the paper measures only. How to reduce peaks via post-processing or architectural changes is a future direction.

8. One-Sentence Summary

The paper's core contribution is reframing the study of "activation outliers": from binary judgment (do outliers exist?) to continuous statistics (what is the actual maximum activation?), and proving this value is a model property rather than a byproduct of scale. When Gemma3-27B-it's peak is 5,260x that of Qwen3.5-35B-A3B, any intuition based on "bigger models are harder to quantize" will mislead you. MoE's 14–23.4x peak reduction, SFT's late-layer compression, and training's monotonic amplification are not footnotes—they are invisible traps deployment engineers face daily. Baidu Research's recommendation is direct: publish maximum activation M and its layer-wise location in model cards. This is not academic advice; it is a production-environment survival guide.

References

• Paper: Measuring Maximum Activations in Open Large Language Models
• Institutions: Shanghai Jiao Tong University, Baidu, Nankai University
• Corresponding authors: Yuchen Li (yuchenli1230@gmail.com), Dawei Yin (yindawei@acm.org)
• arXiv: 2605.15572
• Code: https://github.com/clx1415926/Max_act_llm
• Measurement scale: 27 checkpoints, 8 families, 5,000-sample multi-domain corpus, ~19.5M tokens
• Component coverage: embeddings, hidden states, attention outputs, MLP/MoE outputs, SwiGLU gates, final normalization
• Key results: global max activation 122–696,320 (~4 orders of magnitude); MoE peak reduction 14–23.4x; residual stream carries 22/24 global maxima; training amplifies 1.34x; SFT late-layer compression 1.4x
• Quantization validation: INT8 SQNR co-varies with maximum activation
• Stability: coefficient of variation < 10.1%

*A closing reflection: in AI we have grown used to the "scale decides everything" narrative—more parameters, more data, more training, better models. But the study of activation peaks reveals a counterintuitive truth: some properties do not vary monotonically with scale, and are instead drastically reshaped by micro-choices in architecture and training recipes. Gemma3's 696,320 peak exists not because it is "big" (it is only 27B), but because it made different choices along some design dimension we do not yet fully understand. Future model evaluation may need a "deployment attribute matrix"—beyond accuracy, perplexity, and MMLU, including activation range, quantization friendliness, and memory footprint patterns. These attributes never appear in paper titles, but they appear in every deployment error log. Baidu Research's work fills the first row of that matrix.*