One Block, Multiple Depths: reViT Recurrent Vision Transformers with Depth-Programmable FFN Experts

Paper Overview

• Field: Computer Vision (CV)
• Authors: Adrian Bulat, Yassine Ouali, Georgios Tzimiropoulos
• Published: 2026-10-08
• arXiv: 2610.12448

Summary

This work demonstrates that a single Transformer block, applied recurrently, can match the accuracy of a full-depth vision encoder at comparable inference FLOPs — without intermediate feature distillation.

Key ideas:

• reViT restores depth-specific transformations by representing the FFN at each recurrent depth as a *convex combination* of a small shared expert bank.
• A continuous normalized-depth coordinate programs this mixture, defining a resampleable trajectory through FFN parameter space.

Evaluation and findings:

• Assessed in two regimes: supervised ImageNet-1k training and distillation from a DINOv2 teacher.
• Across both regimes, controlled comparisons identify weight-space merging as the strongest tested MoE family at a matching one-FFN budget, beating token-dispatch and output-mixture alternatives.
• Trained from scratch, reViT-B/16 reaches DeiT III accuracy with roughly 70% fewer stored parameters.
• An 8-expert model distilled only from teacher *output* features nearly retains the full linear-probing accuracy of the DINOv2 teacher and transfers to classification, segmentation, and depth prediction.
• Elastic-depth training allows a single checkpoint to operate at multiple test depths by resampling the same normalized coordinate interval.
• With fixed-depth deployment, the recurrent block can be instantiated as a regular dense graph, eliminating online routing and merging — compute stays the same while deployment storage increases.

Original Abstract (excerpt)

> In this work, we show that a single Transformer block, applied recurrently, can match the accuracy of a full-depth vision encoder at comparable inference FLOPs without intermediate feature distillation. reViT restores depth-specific transformations by representing the FFN at each recurrent depth as a convex combination of a small shared expert bank. A continuous normalized-depth coordinate programs this mixture, defining a resampleable trajectory through FFN parameter space. We evaluate this design in two regimes: supervised ImageNet-1k training and distillation from a DINOv2 teacher. Across both regimes, controlled adaptations identify weight-space merging as the strongest tested MoE family at a matching one-FFN budget, ahead of the token-dispatch and output-mixture alternatives...

*Link*: https://arxiv.org/abs/2610.12448