Your Transformer Is Secretly Elastic: Retraining-Free Elastic Transformers for Any Compute Budget

Forum topic · 小凯 · 2026-10-16

Summary

A 2026 arXiv paper (2610.20588) by Walter Simoncini, Dheeraj Varghese, Gertjan J. Burghouts, and Cees G. M. Snoek introduces a retraining-free method that turns pre-trained Transformers into a single elastic model adaptable across a continuum of computational budgets. The key contribution is a differentiable relaxation of the functional unit-ranking problem: instead of assigning a single importance weight per MLP block or attention head, the method jointly ranks every individual MLP neuron and attention head on a common scale by minimizing soft-pruned model losses under multiple sparsity targets. Evaluated on 27 vision and text Transformer encoders from 22M to 8B parameters, the approach achieves smooth degradation up to 60% sparsity and outperforms one-shot baselines under depth pruning. Pruning DINOv3 ViT-H+/16 to 50% sparsity removes 427M parameters with only a 0.8% drop in ImageNet-1k linear probe accuracy, without weight correction or retraining. At 60% sparsity, a pruned AugReg ViT-B/16 achieves up to 1.80x throughput and 1.26x lower latency on an A100 GPU. Elastic models are generated within minutes, with build time scaling sublinearly with parameter count.

Your Transformer Is Secretly Elastic

Research area: Machine Learning Authors: Walter Simoncini, Dheeraj Varghese, Gertjan J. Burghouts, Cees G. M. Snoek arXiv: 2610.20588

Overview

Scaling up transformers has produced models with broad capabilities, deployed across heterogeneous hardware and fluctuating workloads. Yet model families offer only a few fixed sizes (e.g., S, B, L), forcing users to commit to a model based on hardware and latency constraints. This rigidity is often suboptimal under varying query loads.

The paper introduces a retraining-free method that elastifies pre-trained transformers into a single model that adapts on-the-fly across a continuum of computational budgets. The key contribution is a differentiable relaxation of the functional unit-ranking problem: unlike prior work, which optimizes a single importance weight per MLP block or attention head, this method ranks every individual MLP neuron and attention head jointly, on a common scale, by minimizing the soft-pruned model's loss simultaneously under multiple sparsity targets.

Key Results

Evaluated on 27 vision and text transformer encoders ranging from 22M to 8B parameters.
Achieves smooth degradation up to 60% sparsity and outperforms one-shot baselines under depth pruning.
Pruning DINOv3 ViT-H+/16 to 50% sparsity removes 427M parameters with only a 0.8% drop in ImageNet-1k linear probe accuracy — no weight correction or retraining required.
At 60% sparsity, the pruned AugReg ViT-B/16 achieves up to 1.80x throughput and 1.26x lower single-query latency on an A100 GPU.
Elastic models are generated within minutes, with build time scaling sublinearly with parameter count.