VECA: Elastic Attention Cores for Scalable Vision Transformers

Forum topic · 小凯 · 2026-05-14

Summary

Vision Transformers achieve strong scaling via all-to-all self-attention, but their computational cost grows quadratically with image resolution, limiting use in high-resolution domains. This paper challenges the assumption that pairwise token interactions are necessary for rich visual-semantic representations. The authors propose VECA (Visual Elastic Core Attention), a vision transformer architecture using efficient linear-time core-periphery structured attention built on a small set of learned cores. Cores act as a communication interface: patch tokens exchange information only through core tokens, which are initialized from scratch and propagated across layers. Since N image patches interact only with a fixed number C of learned core embeddings, complexity is linear O(N), bypassing quadratic scaling. Unlike prior cross-attention architectures, VECA maintains and iteratively updates the full set of N input tokens, avoiding a C-way bottleneck. Combined with nested training along the core axis, models can elastically trade compute for accuracy at inference. On classification and dense tasks, VECA matches recent vision foundation models at reduced compute. Authors include Alan Z. Song, Deva Ramanan, Michael J. Tarr, and Andrew F. Luo; arXiv: 2605.12491.

Paper Overview

Research Area: Computer Vision (CV) Authors: Alan Z. Song, Yinjie Chen, Mu Nan, Rui Zhang, Jiahang Cao, Weijian Mai, Muquan Yu, Hossein Adeli, Deva Ramanan, Michael J. Tarr, Andrew F. Luo Published: 2026-05-12 arXiv: 2605.12491

Abstract

Vision Transformers (ViTs) achieve strong data-driven scaling by leveraging all-to-all self-attention. However, this flexibility incurs a computational cost that scales quadratically with image resolution, limiting ViTs in high-resolution domains. Underlying this approach is the assumption that pairwise token interactions are necessary for learning rich visual-semantic representations.

In this work, the authors challenge this assumption, demonstrating that effective visual representations can be learned without any direct patch-to-patch interaction. They propose VECA (Visual Elastic Core Attention), a vision transformer architecture that uses efficient linear-time core-periphery structured attention enabled by a small set of learned cores.

Key Ideas

Cores as a communication interface: patch tokens exchange information only through these core tokens; the cores are initialized from scratch and propagated across layers.
Linear complexity: N image patches interact directly only with a predetermined number C of learned core embeddings, yielding O(N) complexity and bypassing the quadratic scaling of full self-attention.
Avoiding the C-way bottleneck: compared with prior cross-attention architectures, VECA maintains and iteratively updates the complete set of N input tokens.
Elastic inference: nested training along the core axis allows models to elastically trade computation for accuracy at inference time.
Results: on classification and dense prediction tasks, VECA achieves performance comparable to recent vision foundation models at reduced computational cost.

---

*Auto-collected on 2026-05-14. Original discussion: zhichai.net*