IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression

Forum topic · 小凯 · 2026-05-19

Summary

IO-SVD is an SVD-based low-rank compression method for large language models that improves on prior input-whitening approaches by whitening both the input and output sides of each weight matrix. Input whitening uses activation covariance statistics to allocate more rank to directions with high activation variance, while output whitening builds a sensitivity measure from a second-order expansion of a KL divergence loss, capturing how compressing a direction in one layer shifts the downstream output distribution. After decomposition in this dual-whitened space, singular values across all layers are ranked globally by their contribution to a first-order calibration loss, so the compression budget is spent where it hurts least instead of applying a uniform rank reduction. The method also integrates with quantization: rows of the decomposed matrices are selected for 8-bit quantization based on predicted loss impact, keeping sensitive rows at higher precision. Open questions include the accuracy of the second-order KL approximation at high compression ratios and the hardware efficiency of the resulting matrix multiplications.

LLMs are too large to fit in memory on edge devices. SVD low-rank decomposition is a hardware-agnostic compression approach: a large weight matrix is factored into two smaller matrices whose product approximates the original operation with fewer parameters. But existing methods have two shortcomings: they use the input-whitened space to determine the decomposition, ignoring information on the output side; and they apply a uniform compression ratio across all layers, without knowing which layers are important and which are insensitive.

IO-SVD, proposed by Abbasi, Thrash, Qin, Pirsiavash, and Kolouri, improves both ends.

Input whitening: compute the covariance of activations to identify directions where activation variance is large; these directions retain more rank.

Output whitening: construct an output-side sensitivity measure from a second-order expansion of a KL loss (change in the model's output distribution) — if you compress a direction in one layer, how far does the next layer's distribution drift? This is more precise than using norms alone.

After SVD in this dual-whitened space, each layer's singular values are evaluated with a first-order calibration loss — rather than uniformly cutting 50% everywhere, singular components are globally ranked by their contribution to the loss, and the least contributing ones are removed under a total budget constraint.

A hybrid improvement combines this with quantization: which rows of the decomposed small matrices get 8-bit quantization is also decided based on predicted loss change — insensitive rows are quantized, sensitive rows stay at higher precision.

Open questions:

Approximation quality of the second-order KL expansion: if the output distribution changes a lot after compression (high compression ratios), does the second-order approximation remain accurate?
Hardware support for the hybrid SVD–quantization scheme: matrix multiplications after SVD may be less efficient than the original dense matrix on some hardware — does the paper's claimed "practical inference speedup" require specialized kernel implementations?

References

1. Abbasi, A., Thrash, C., Qin, H., Pirsiavash, H., & Kolouri, S. (2026). *IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression*. arXiv:2605.15626 [cs.LG]. 2. Hsu, Y., et al. (2022). *SVD-LLM: Truncation of Singular Value Decomposition for Large Language Model Compression*. NeurIPS. 3. Frantar, E., et al. (2023). *GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers*. ICLR.

IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression

Summary

References

Tags