论文概要
Research area: Computer Vision Authors: Alan Z. Song, Yinjie Chen, Mu Nan, Rui Zhang, Jiahang Cao, Weijian Mai, Muquan Yu, Hossein Adeli, Deva Ramanan, Michael J. Tarr, Andrew F. Luo Published: 2026-05-12 arXiv: 2605.12491
Summary
Vision Transformers (ViTs) achieve strong data-driven scaling by leveraging all-to-all self-attention. However, this flexibility incurs a computational cost that scales quadratically with image resolution, limiting ViTs in high-resolution domains. Underlying this approach is the assumption that pairwise token interactions are necessary for learning rich visual-semantic representations.
This work challenges that assumption, demonstrating that effective visual representations can be learned without any direct patch-to-patch interaction. The authors propose VECA (Visual Elastic Core Attention), a vision transformer architecture that uses efficient linear-time core-periphery structured attention enabled by a small set of learned cores.
Key ideas
- Cores as a communication interface: patch tokens exchange information only through a small set of learned core tokens, which are initialized from scratch and propagated across layers.
- Linear complexity: since N image patches interact directly only with a predetermined number C of learned "core" embeddings, complexity is O(N), bypassing quadratic scaling.
- No C-way bottleneck: compared with prior cross-attention architectures, VECA maintains and iteratively updates the full set of N input tokens.
- Elastic inference: nested training along the core axis allows the model to elastically trade off computation and accuracy at inference time.
Results
On classification and dense tasks, VECA achieves performance comparable to state-of-the-art vision foundation models while reducing computational cost.
--- *Auto-collected on 2026-05-14*