TRecViT: Google DeepMind's Recurrent Video Transformer Hits 300 FPS with Causal Modeling

Forum topic · 小凯 · 2026-05-22

Summary

TRecViT is a recurrent video transformer from Google DeepMind (arXiv: 2412.14294) designed to overcome the O(T^2) compute blow-up that plagues conventional video transformers. The architecture factorizes video into three dimensions: spatial self-attention within each frame, temporal processing via a Gated Linear Recurrent Unit (Gated LRU) that compresses history into a recurrent state, and channel mixing with MLPs. Because it is causal—processing frames strictly in order without peeking at future frames—it achieves O(1) per-frame inference cost regardless of clip length, making it suitable for real-time robotics, autonomous driving, and AR workloads. Compared with ViViT-L, TRecViT reportedly uses roughly one-third the parameters, about one-twelfth the memory at 64 frames, and reaches up to 300 FPS inference—far above real-time 24 FPS video. The post presents TRecViT as evidence that elegant recurrent-attention hybrids, rather than brute-force GPU scaling, can unlock efficient long-video understanding.

Introduction: The Time Problem in Video AI

Video has always been the hardest modality for AI models to handle. A single image can be processed in one pass, but a one-minute video is effectively thousands of stacked frames. Traditional video transformers apply global attention across all frames, so compute grows quadratically with sequence length — the notorious $$O(T^2)$$ complexity. Longer videos mean memory and compute explode.

TRecViT (A Recurrent Video Transformer), from Google DeepMind, takes a different route: instead of brute-forcing compute, it uses a factorized architecture that reportedly reaches 300 FPS on ordinary hardware.

> Note: $$O(T^2)$$ complexity > In standard self-attention, compute scales with the square of sequence length $$T$$ . Doubling the number of frames quadruples the cost, quickly exhausting memory.

---

1. Factorizing Space, Time, and Channels

TRecViT splits video into three dimensions and handles each separately:

> Note: Gated LRU (Gated Linear Recurrent Unit) > A neural component combining linear dynamical systems with gating. It maintains long-range memory, supports fast parallel training, and delivers very low inference latency.

---

2. Causal Modeling: Only the Present Matters

Conventional video AI is non-causal: it must watch an entire clip before producing an answer. For autonomous driving or robotics, waiting is not an option.

TRecViT is built for causal video modelling: it makes a judgment after each frame and never depends on future frames. Thanks to the LRU, per-frame inference compute is constant:

$Compute_{per\_frame} = O(1)$

A ten-second clip or a two-hour film — each frame costs the same.

---

3. The Efficiency Ledger

Compared with same-class giants like ViViT-L:

1. Parameters: reduced to about one-third. 2. Memory: at 64 frames, roughly one-twelfth the footprint — and the savings grow with clip length. 3. Speed: up to 300 FPS inference, more than 10x human viewing speed (24 FPS).

The authors position it as efficient enough to run on lightweight AR headsets, not just data-center GPUs.

---

4. Conclusion: Geometry over Brute Force

TRecViT is presented as evidence that stacking GPUs is not the only path forward. By respecting the causal flow of time and using elegant mathematical structures (like the Gated LRU) to dissolve the compute bottleneck, hybrid attention-recurrence architectures offer a scalable route to long-video understanding — a return to architectural efficiency over raw compute.

---

References

Core paper:

Title: TRecViT: A Recurrent Video Transformer
arXiv ID: 2412.14294
Institution: Google DeepMind
Code: [google-deepmind/trecvit]

Keywords: Gated LRU, Causal Video Modelling,

\(O(1)\)

inference complexity, 300 FPS.