Super Weights in LLMs and the Failure of Selective Training (arXiv 2507.08699)

Forum topic · 小凯 · 2026-07-13

Summary

A new arXiv paper (2507.08699) by Shreyas Subramanian, Adewale Akinfaderin, and Akarsha Sehwag examines Super Weights—individual parameters whose removal reportedly degrades LLM performance by orders of magnitude. The authors show this pruning-induced degradation does not universally apply to all LLMs, and that Super Weight-aware training is ineffective. Training Super Weights in isolation (100 to 8,192 parameters) drops accuracy to random-guessing levels on OLMo-1B and OLMo-7B, and expanding to neighborhoods of up to 36K parameters does not help. The failure is specific to Super Weight coordinates: training an equal number of random positions in the same down_proj layers improves over baseline, showing the collapse comes from targeting Super Weights rather than sparsity itself. Ordinary LoRA succeeds using only 0.16% of parameters via low-rank updates, and ablations across 10 random seeds confirm that restricting LoRA updates to Super Weight coordinates yields statistically indistinguishable results. The findings establish that parameter importance does not imply trainability, and effective fine-tuning relies on structured decomposition of entire layers rather than targeting individual important weights.

Paper Overview

Field: Machine Learning Authors: Shreyas Subramanian, Adewale Akinfaderin, Akarsha Sehwag Published: 2025-07-12 arXiv: 2507.08699

Summary

Recent work identified Super Weights, individual parameters whose removal degrades model performance by orders of magnitude. The authors show that this degradation due to pruning Super Weights does not universally apply to all LLMs. Furthermore, if these parameters are so important, Super Weight-aware training should be effective. They show the opposite.

Key Findings

Training Super Weights in isolation (100 to 8,192 parameters) drops accuracy to random-guessing levels on both OLMo-1B and OLMo-7B, and expanding to local neighborhoods of up to 36K parameters provides no improvement.
The failure is specific to Super Weight coordinates: training an equal number of randomly chosen positions in the same down_proj layers instead improves over the baseline, so the collapse comes from targeting Super Weights, not from sparsity itself.
Ordinary LoRA succeeds by updating every position in the attention weight matrices through low-rank structure, using only 0.16% of parameters; applying the same low-rank updates to down_proj also succeeds.
Ablations over 10 random seeds confirm that restricting LoRA updates to positions corresponding to Super Weight coordinates produces statistically indistinguishable results.

Conclusion

These findings establish that parameter importance does not imply the feasibility of training that parameter in isolation. Effective fine-tuning relies on structured decomposition of the entire layer rather than targeting individual important weights.

---

*Auto-collected on 2026-07-13.*

Super Weights in LLMs and the Failure of Selective Training (arXiv 2507.08699)

Summary

Paper Overview

Summary

Key Findings

Conclusion

Tags