TRecViT Deep Dive: How Google DeepMind's Recurrent Video Transformer Hits 300 FPS

Forum topic · 小凯 · 2026-05-22

Summary

TRecViT (A Recurrent Video Transformer), developed by Google DeepMind, is a causal video modeling architecture that avoids the O(T^2) complexity of traditional video Transformers. Instead of comparing all frames jointly, it factorizes video processing into three dimensions: spatial self-attention within each frame, temporal modeling via a Gated Linear Recurrent Unit (LRU) that compresses history into a recurrent state, and channel-wise MLPs for feature fusion. Because each frame is processed causally—depending only on past frames—the per-frame compute is O(1) regardless of clip length, making it suitable for real-time robotics, autonomous driving, and AR applications. Compared with heavyweight baselines like ViViT-L, TRecViT reportedly uses roughly one-third the parameters and about one-twelfth the memory at 64 frames, while reaching around 300 FPS inference—well beyond human playback speeds of 24 FPS. The paper (arXiv: 2412.14294) and code are open-sourced by Google DeepMind. The architecture illustrates a broader trend: replacing brute-force attention over long temporal sequences with elegant recurrent structures that respect the causal flow of time.

TRecViT: A Recurrent Video Transformer — When AI Video Learns to Fight Speed with Speed

Foreword: The Long River of Time and the Shackles of Compute

Since AI entered the video domain, it has faced one fundamental difficulty: time.

A model can grasp an image at a glance, but a one-minute video means thousands of stacked frames. Traditional Transformers insist on performing "global comparison" across all frames together. As videos get longer, computation balloons like a snowball — the infamous $$O(T^2)$$ complexity curse.

It is like asking someone to record the trajectory of every drop of water in a long river — a task doomed to exhaust them.

In response, Google DeepMind introduced TRecViT (A Recurrent Video Transformer). Instead of brute-forcing compute, it performs a kind of "bone-shrinking art," reaching 300 FPS even on ordinary devices.

> Note: $$O(T^2)$$ complexity > In standard self-attention, compute scales with the square of sequence length $$T$$ . Doubling the number of frames quadruples the cost, quickly exhausting memory.

1. Factorizing Space and Time: A Perfect Pairing

TRecViT's speed comes from decomposition. It splits video information into three independent dimensions: time, space, and channels.

> Note: Gated LRU (Gated Linear Recurrent Unit) > A neural component combining linear dynamical systems with gating mechanisms. It maintains long-range memory while enabling fast parallel training and very low inference latency.

2. The Power of Causality: Live in the Present

Traditional video AI acts as a "Monday-morning quarterback" — it must watch an entire clip before producing an answer. Such non-causal models are dangerous for autonomous driving or robotics, where the car would have crashed before the model finished thinking.

TRecViT adopts causal video modelling: time's arrow moves only forward. It judges each frame immediately, never relying on future frames. Thanks to the LRU, its inference cost per frame is constant at $$O(1)$$ — whether the input is a ten-second clip or a two-hour film.

Compute_{per\_frame} = O(1)

3. The Astonishing "Slimming" Ledger

Compared with same-tier giants like ViViT-L:

1. Lightweight: parameter count drops to about one-third. 2. Lean memory: memory usage (at 64 frames) falls to about one-twelfth — savings grow with longer clips. 3. Lightning fast: inference reaches 300 FPS, more than ten times human playback speed (24 FPS).

It is like a "studio apartment" for video AI: no ornate decoration, but maximum space efficiency — light enough to run smoothly even on lightweight AR headsets.

4. Conclusion: Beyond Compute Supremacy, Back to Structural Elegance

TRecViT is a wake-up call to approaches that simply stack GPU compute. In the vast ocean of video data, brute force has limits; the better path is to follow the causal flow of time and resolve computational bottlenecks with elegant mathematical structures like the LRU. This is not just a performance leap, but a return to geometric and physical beauty in AI architecture design.

References

Paper: TRecViT: A Recurrent Video Transformer
arXiv: 2412.14294
Institution: Google DeepMind
Code: [google-deepmind/trecvit]

Key technical terms: Gated LRU, Causal Video Modelling,

\(O(1)\)

per-frame inference, 300 FPS real-time processing.