Super Weights in LLMs and the Failure of Selective Parameter Training

Forum topic · 小凯 · 2026-07-13

Summary

A recent arXiv paper (2507.08699) by Shreyas Subramanian, Adewale Akinfaderin, and Akarsha Sehwag investigates Super Weights — individual parameters whose removal degrades LLM performance by orders of magnitude. The authors show this degradation does not apply universally across all LLMs. Experiments on OLMo-1B and OLMo-7B demonstrate that training Super Weights in isolation (between 100 and 8,192 parameters) drops accuracy to random-guessing levels. Notably, this failure is specific to Super Weight coordinates: training the same number of randomly chosen positions in the same down_proj layers actually improves over the baseline. In contrast, vanilla LoRA, which updates every position in attention weight matrices via low-rank structure, succeeds using only 0.16% of parameters. The findings establish that parameter importance does not imply parameter trainability in isolation, and that effective fine-tuning relies on structured decomposition across entire layers rather than selective training of critical individual weights.

Paper Overview

Field: LLM Training Authors: Shreyas Subramanian, Adewale Akinfaderin, Akarsha Sehwag Published: 2025-07-12 arXiv: 2507.08699

Abstract (Translated)

Recent work identified Super Weights, individual parameters whose removal degrades model performance by orders of magnitude. We show that this degradation does not universally apply to all LLMs. Training Super Weights in isolation (100 to 8,192 parameters) drops accuracy to random-guessing levels on both OLMo-1B and OLMo-7B. The failure is specific to Super Weight coordinates: training an equal number of randomly chosen positions in the same down_proj layers instead improves over the baseline. Vanilla LoRA, updating every position in attention weight matrices through low-rank structure, succeeds with only 0.16% of parameters. These findings establish that parameter importance does not imply parameter trainability in isolation, and that effective fine-tuning relies on structured decomposition of entire layers.

Key Findings

Super Weights degrade performance when removed, but this does not universally apply across all LLMs.
Isolated training of Super Weights fails: updating only 100 to 8,192 Super Weight parameters drops accuracy to random-guessing levels on both OLMo-1B and OLMo-7B.
The failure is coordinate-specific: training an equal number of randomly chosen positions in the same down_proj layers improves over the baseline, so the problem lies with the Super Weight coordinates themselves.
LoRA succeeds: vanilla LoRA updates every position in attention weight matrices through low-rank structure and works with only 0.16% of parameters.
Conclusion: parameter importance does not imply parameter trainability in isolation; effective fine-tuning depends on structured decomposition of entire layers.

--- *Auto-collected on 2025-07-13*