[论文] Your Transformer Is Secretly Elastic

论文概要

研究领域: ML 作者: Walter Simoncini, Dheeraj Varghese, Gertjan J. Burghouts, Cees G. M. Snoek 发布时间: 2026-10-14 arXiv: 2610.20588

中文摘要

Transformer的规模化产生了具有广泛能力的模型，部署在异构硬件和波动的工作负载上。然而，模型家族仅提供少数固定尺寸（如S、B、L），迫使用户基于硬件和延迟约束做出选择。这种刚性在变化的查询负载下往往并非最优。我们引入一种无需重训练的方法，将预训练Transformer弹性化为单一模型，可在连续计算预算范围内自适应调整。我们的核心贡献是函数单元排序问题的可微松弛：与以往工作不同（它们仅为每个MLP块或注意力头优化单一重要性权重），我们在共同尺度上联合排序每个单独的MLP神经元和注意力头，通过同时在多个稀疏目标下最小化软剪枝的模型损失来实现。在27个从22M到8B参数的视觉和文本Transformer编码器上，我们的方法在高达60%稀疏度下实现平滑退化，并在深度剪枝下优于一次性基线。例如，将DINOv3 ViT-H+/16剪枝到50%稀疏度可移除4.27亿参数，而ImageNet-1k线性探测准确率仅下降0.8%，无需权重校正或重训练。在60%稀疏度下，剪枝后的AugReg ViT-B/16模型在A100 GPU上实现高达1.80倍的吞吐量和1.26倍的更低单查询延迟。弹性模型在几分钟内生成，构建时间随参数数量亚线性增长。

原文摘要

Scaling up transformers has produced models with broad capabilities, deployed across heterogeneous hardware and fluctuating workloads. Yet, model families offer only a few fixed sizes (e.g., S, B, L), forcing users to commit to a model based on hardware and latency constraints. This rigidity is often suboptimal under varying query loads. We introduce a retraining-free method that elastifies pre-trained transformers into a single model that adapts on-the-fly across a continuum of computational budgets. Our key contribution is a differentiable relaxation of the functional unit-ranking problem: unlike prior work, which optimizes a single importance weight per MLP block or attention head, we rank every individual MLP neuron and attention head jointly, on a common scale, by minimizing the model...

#论文 #arXiv #ML #小凯