Large language models are too big to fit into the memory of edge devices. SVD low-rank decomposition is a hardware-agnostic compression approach — it factorizes a large weight matrix into the product of two smaller matrices, approximating the original operation with fewer parameters. But existing methods have two drawbacks: they determine the decomposition using the input whitening space only, ignoring information on the output side, and they apply a uniform compression ratio across all layers without knowing which layers are important and which are insensitive.
IO-SVD, proposed by Abbasi, Thrash, Qin, Pirsiavash, and Kolouri, improves on both ends.
Input whitening: the covariance of activations is computed statistically to identify directions where activation variation is large; these directions retain more rank.
Output whitening: a second-order expansion of a KL loss (change in the model's output distribution) is used to construct an output-side sensitivity measure — if you compress a certain direction of a layer, how far will the next layer's distribution drift? This is more precise than relying purely on norms.
Adaptive-rank truncation: after performing SVD in this dual whitening space, the singular values of each layer are evaluated with a first-order calibration loss — rather than uniformly cutting 50% everywhere, singular components are globally ranked by their contribution to loss across all layers, and the least contributing ones are removed under a total budget constraint.
Combined with quantization: the method also includes an improvement that combines SVD with quantization: choosing which rows of the decomposed small matrices to quantize to 8-bit is also decided based on predicted loss change — insensitive rows get quantized, sensitive rows keep high precision.
Open questions:
- Quality of the second-order KL expansion approximation — if the model's output distribution changes drastically after compression (at high compression ratios), does the second-order approximation remain accurate?
- Hardware support for the mixed SVD-quantization scheme — matrix multiplications after SVD may be less efficient than the original matrices on some hardware; does the paper's claimed "actual inference speedup" require specialized kernel implementations?
References
1. Abbasi, A., Thrash, C., Qin, H., Pirsiavash, H., & Kolouri, S. (2026). *IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression*. arXiv:2605.15626 [cs.LG]. 2. Hsu, Y., et al. (2022). *SVD-LLM: Truncation of Singular Value Decomposition for Large Language Model Compression*. NeurIPS. 3. Frantar, E., et al. (2023). *GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers*. ICLR.