English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original
智柴网 中文原文

Super Weights in LLMs: Why Training Critical Parameters in Isolation Fails

Forum topic · 小凯 · 2026-07-13

Summary

A recent arXiv paper (2507.08699) by Subramanian, Akinfaderin, and Sehwag examines Super Weights—individual parameters whose removal degrades LLM performance by orders of magnitude—and shows that their importance does not translate into trainability. When the authors fine-tune only Super Weights (100 to 8,192 parameters) in OLMo-1B and OLMo-7B, accuracy collapses to random-guessing levels. Notably, this failure is coordinate-specific: training the same number of randomly chosen positions in the identical down_proj layers actually improves performance over the baseline. In contrast, vanilla LoRA, which updates every position in attention weight matrices via low-rank structure, succeeds using just 0.16% of parameters. The findings establish that parameter importance does not imply parameter trainability in isolation, and that effective fine-tuning depends on structured decomposition across entire layers rather than targeting individually critical weights.

Overview

Field: LLM training Authors: Shreyas Subramanian, Adewale Akinfaderin, Akarsha Sehwag Published: 2025-07-12 arXiv: 2507.08699

Summary

Recent work identified Super Weights, individual parameters whose removal degrades model performance by orders of magnitude. This paper shows that this degradation does not universally apply to all LLMs, and—more importantly—that the importance of these weights does not make them viable fine-tuning targets.

Key findings

  • Training Super Weights in isolation (100 to 8,192 parameters) drops accuracy to random-guessing levels on both OLMo-1B and OLMo-7B.
  • The failure is specific to Super Weight coordinates: training an equal number of randomly chosen positions in the same down_proj layers instead improves over the baseline.
  • Vanilla LoRA, which updates every position in attention weight matrices through low-rank structure, succeeds using only 0.16% of parameters.

Conclusion

These findings establish that parameter importance does not imply parameter trainability in isolation. Effective fine-tuning relies on structured decomposition across entire layers rather than targeting individually critical weights, even ones as influential as Super Weights.

---

Paper: arXiv:2507.08699

Tags

#llm-training#super-weights#lora#fine-tuning#olmo#parameter-importance#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178379432