VECA: How Core-Node Attention Makes Vision Transformers Scale Linearly

Forum topic · 小凯 · 2026-05-13

Summary

VECA (Visual Elastic Core Attention) is a new architecture that replaces the O(N²) self-attention in Vision Transformers with a core-periphery design, reducing compute to O(N·C) where C is a small, fixed number of learnable core embeddings. By removing direct patch-to-patch interactions and routing information through shared cores, VECA achieves linear scaling with image resolution, yielding up to 256× speedup at 2048×2048 while remaining competitive on ImageNet classification and dense tasks like segmentation and detection. The cores are learned end-to-end, propagated across all layers, and the full patch sequence is preserved to avoid information loss. A nested training strategy enables elastic inference, letting users trade accuracy for compute on demand. The work challenges the assumption that all-to-all attention is necessary for strong visual representations.

Key Points

The quadratic bottleneck: Standard Vision Transformer self-attention costs O(N²) in patch count N, making high-resolution images (medical scans, satellite imagery, autonomous driving) prohibitively expensive. Going from 224×224 to 2048×2048 increases compute ~7,000×.
Core-periphery design: VECA abandons direct patch-to-patch attention. Instead, N patches communicate only through C learnable core embeddings, and cores exchange information among themselves. Total cost is O(N·C) + O(C²) ≈ O(N) when C is a constant.
Why cores are enough: Cores are randomly initialized and trained, not hand-designed. They propagate across all layers, progressively refining their role. The full N-token sequence is preserved at every step, unlike Perceiver-style bottlenecks that compress the input.
Empirical results: VECA matches competitive vision foundation models on ImageNet classification and maintains strong performance on dense prediction tasks (segmentation, detection), confirming that no essential spatial information is lost.
Elastic inference: A nested training scheme along the core axis lets a single trained model run at multiple compute/accuracy points—fewer cores for speed, more cores for precision—without retraining.
Conceptual impact: The paper challenges the long-held belief that all-to-all pairwise attention is required for rich visual representations, opening a design space for alternative interaction topologies (graph, hierarchical, dynamic routing).

Detailed Overview

The Problem

Vision Transformer (ViT) splits an image into patches (e.g., 16×16 pixels) and applies self-attention across all patches. Self-attention produces an N×N attention matrix, costing O(N²). At 1024×1024 with 16×16 patches, N=4,096 and compute is ~16.8M operations; at 2048×2048, N=16,384 and compute balloons to ~268M. This quadratic scaling is the structural weakness preventing ViT from serving high-resolution vision domains.

The VECA Solution

VECA introduces C learnable core embeddings that act as communication hubs:

Patches attend only to cores (not to each other).
Cores attend to each other.
Information flows: Patch A → Core X → Core Y → Patch B.

Because C is a small constant (e.g., 64 or 128) independent of image resolution, the dominant term O(N·C) scales linearly with N. At 2048×2048 resolution, this yields roughly a 256× speedup over standard self-attention.

Why Information Is Not Lost

Three design choices preserve representational power:

1. Learned cores: Cores emerge through training to specialize (e.g., aggregating sky, foreground, or contour regions). 2. Cross-layer propagation: Cores are passed from layer to layer, allowing progressive refinement rather than independent re-learning each block. 3. Full sequence retention: All N patches remain in the representation; only the *interaction pattern* changes, not the token count. This is critical for dense tasks that need per-pixel precision.

Experimental Performance

On ImageNet, VECA achieves accuracy competitive with leading vision foundation models at substantially lower compute. On dense tasks (semantic segmentation, object detection), performance remains strong, validating that the core-routed interaction preserves spatial detail.

Elastic Inference

Nested training exposes a trade-off axis: the number of active cores. A single model can be deployed at multiple operating points—mobile devices use fewer cores for speed and battery efficiency, servers use more cores for accuracy—without retraining or storing separate weights.

Broader Significance

VECA exemplifies a core-periphery topology seen throughout complex systems—brain networks, urban transit, social networks, ecosystems—where efficient flow does not require full connectivity. By demonstrating that direct patch-to-patch interaction is *not* necessary for learning effective visual representations, the work reframes a foundational assumption of Transformer design and invites exploration of alternative interaction graphs (graph attention, hierarchical attention, dynamic routing).

Reference

Song, A. Z., Chen, Y., & Nan, M. (2025). *Elastic Attention Cores for Scalable Vision Transformers*. arXiv preprint.

Related Work

ViT (Dosovitskiy et al., 2020) — original Vision Transformer
Attention Is All You Need (Vaswani et al., 2017) — the Transformer foundation
Perceiver (Jaegle et al., 2021) — latent-array cross-attention with input compression
Set Transformer (Lee et al., 2019) — inducing-point attention
Linear Attention (Katharopoulos et al., 2020) — kernel-based O(N) attention