SliderQuant: Adaptive Layer-wise Quantization for Accurate LLM Compression

Overview

SliderQuant is a post-training quantization (PTQ) framework, accepted at ICLR 2026, that adapts its compression strategy layer by layer instead of using a single global configuration. The paper introduces a "sliding" mechanism that varies quantization aggressiveness across the network depth, plus learnable sliding parameters that are co-optimized with the model.

Key points

• Layer sensitivity is non-uniform. Empirical analysis shows that the first and last transformer blocks suffer the largest quantization errors. Shallow layers act as the model's perceptual front-end and lack upstream buffering, while the final layer directly produces output logits with no downstream correction. Middle layers carry more redundancy and tolerate aggressive low-bit quantization.
• Three sliding-window regimes. SliderQuant applies a conservative strategy at shallow layers (higher bit-width or finer-grained grouping), an aggressive strategy in the middle (lower bit-width, coarser granularity), and a refined strategy at deep layers to preserve output quality.
• Progressive intra-layer quantization. Within each layer, precision is reduced gradually—starting from a higher baseline, stepping down toward the target bit-width, and ending with calibration—so the model adapts smoothly rather than absorbing a one-shot precision shock.
• Learnable sliding parameters. Window size, transition smoothness, and per-group adjustments are parameterized and trained end-to-end alongside the task loss, allowing the model to discover its own layer-wise quantization schedule instead of relying on hand-tuned heuristics.
• Mathematical formulation. For layer l, a sensitivity coefficient is computed as alpha_l = f(l; theta_slide) with a learnable theta_slide, and the per-layer bit-width becomes bit_width_l = base_bit_width + alpha_l * delta_bits. Standard quantization/dequantization uses per-layer scale s_l and zero-point z_l that are themselves learned.
• Broad empirical coverage. Experiments span Llama, Llama2, Llama3, Qwen2.5, DeepSeek-R1 distilled variants, and large MoE architectures, evaluated on zero-shot common-sense reasoning, math, and code generation. SliderQuant outperforms prior PTQ baselines, including rotation-based methods, and remains robust under challenging weight-activation joint quantization.
• Practical impact. Because PTQ avoids costly retraining, the approach makes it feasible to deploy 70B-scale models with substantially reduced memory and compute footprints while preserving accuracy, supporting LLM deployment on commodity hardware.

Outlook

The authors highlight dynamic input-dependent quantization, extension to multimodal architectures where different encoders have distinct sensitivity profiles, and hardware-aware sliding that navigates the accuracy-efficiency Pareto frontier on specific accelerators as promising future directions.

References

1. Wang, S., Li, C., Kang, Y., Fan, J., Ou, Z., & Yao, A. (2026). *SliderQuant: Accurate Post-Training Quantization for LLMs*. https://arxiv.org/abs/2603.25284 2. Frantar, E., et al. (2022). *GPTQ: Accurate post-training quantization for generative pre-trained transformers*. ICLR 2023. 3. Xiao, G., et al. (2023). *SmoothQuant: Accurate and efficient post-training quantization for large language models*. ICML 2023. 4. Dettmers, T., et al. (2022). *LLM.int8(): 8-bit matrix multiplication for transformers at scale*. NeurIPS 2022. 5. Lin, J., et al. (2023). *AWQ: Activation-aware weight quantization for LLM compression and acceleration*. MLSys 2024. 6. Ashkboos, S., et al. (2024). *Quarot: Outlier-free 4-bit inference in rotated LLMs*. NeurIPS 2024. 7. Tseng, C.-Y., et al. (2024). *Rotation and permutation for advanced quantization of large language models*. https://arxiv.org/abs/2406.01721