LiteFrame: 71% Smaller Video Encoder from Google DeepMind Unlocks 8x More Frames for Video LLMs

Forum topic · 小凯 · 2026-06-20

Summary

LiteFrame is a lightweight video encoder from Google DeepMind and Seoul National University that addresses an overlooked bottleneck in video LLM efficiency: while most methods compress visual tokens only at the LLM side, the ViT encoder itself still processes every frame with a 304M-parameter ViT-Large. LiteFrame is an 87M-parameter student encoder (71% smaller) that internalizes spatiotemporal token compression directly into the visual backbone. Its key innovation, Compressed Token Distillation (CTD), trains the student to align with the teacher's WAP-compressed (weighted average pooling) information-dense representations rather than raw dense outputs. The architecture interleaves ViT-Base layers with depthwise separable temporal convolutions (DWTempConv) with temporal striding. Built on InternVL3-8B, LiteFrame processes 8x more input frames while cutting end-to-end latency by 28-35%, improving accuracy by 0.4-2.1 points, achieving 12.7x faster LLM prefill at 64 frames and 5.3x faster ViT encoding. It outperforms posterior compression methods (ToMe, PruMerge, FastVID) and efficient encoders (FastVLM, VideoPanda), and surpasses AutoGaze on high-resolution benchmarks without high-resolution training. Paper: arXiv:2605.17260.

LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs

> Translated and adapted from a zhichai.net forum post. Paper: "LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs" by Jihwan Kim, Nikhil Parthasarathy, Danfeng Qin, Junhwa Hur, Deqing Sun, Bohyung Han, Ming-Hsuan Yang, Boqing Gong (Google DeepMind, Seoul National University). arXiv: 2605.17260. Project page: https://jjihwan.github.io/projects/LiteFrame

Key points

The blind spot: Video LLM efficiency work focuses on compressing visual tokens at the LLM side, but the ViT encoder (e.g., 304M ViT-Large) still runs per-frame on every frame. As LLM latency drops, ViT latency becomes the dominant bottleneck — end-to-end latency barely improves despite posterior compression (ToMe, PruMerge, FastVID).
LiteFrame: an 87M-parameter lightweight video encoder (71% smaller than the teacher ViT-Large-24L) that internalizes spatiotemporal token compression into the visual backbone, processing 8x more frames with 28–35% lower end-to-end latency and higher accuracy.
Key innovations:
WAP (Weighted Average Pooling): an attention-weighted pooling operator that compresses tokens 16x and outperforms average/max pooling, subsampling, ToMe, PruMerge, and FastVID as a compression primitive (avg 62.0 on 4 benchmarks vs 59.7–61.2 for baselines).
CTD (Compressed Token Distillation): the student aligns with the teacher's *compressed*, information-dense representations via L_CTD(θ) = ||S_θ(x) − P(T(x))||², instead of the teacher's redundant dense outputs. This internalizes WAP's attention-weighting into the student's parameters.
DWTempConv: depthwise separable 1D temporal convolutions interleaved with spatial attention, with temporal stride-2 layers after blocks 4 and 8. At 256 frames: 174.84ms / 17.92 TFLOPs / 87.15M params — best latency and FLOPs among tested architectures (<1M parameter overhead vs plain ViT-Base-12L).
LMA (Language Model Adaptation): LoRA (r=4, α=8, dropout=0.05) fine-tuning with 128–512 frame sampling at 1–4 FPS.

Headline results (InternVL3-8B baseline)

| Baseline | Method | Frames | Total latency | Avg accuracy | |---|---|---|---|---| | 8 frames | InternVL3-8B | 8 | 208.4 ms | 59.0% | | 8 frames | +FastVID | 32 (4x) | 224.8 ms (+7.9%) | 59.5% | | 8 frames | +LiteFrame | 64 (8x) | 150.1 ms (−28.0%) | 61.1% (+2.1) | | 16 frames | +FastVID | 64 (4x) | 406.2 ms (+0.6%) | 59.5% (−2.7) | | 16 frames | +LiteFrame | 128 (8x) | 272.6 ms (−32.5%) | 63.7% (+1.5) | | 32 frames | +FastVID | 128 (4x) | 794.9 ms (−2.4%) | 62.9% (−2.4) | | 32 frames | +LiteFrame | 256 (8x) | 532.3 ms (−34.6%) | 65.7% (+0.4) |

12.7x faster LLM prefill at 64 frames; 5.3x faster ViT encoding.
vs AutoGaze (NVILA-8B-Video): scaling 32→256 frames, AutoGaze's total latency grows +682.1% (6106.5 ms) with zero accuracy gain, while LiteFrame is −34.6% (532.3 ms) with +0.4% accuracy.
vs efficient encoders (32 frames): LiteFrame (91.9 ms, 58.0%) beats FastVLM (231.5 ms, 47.6%) and VideoPanda (382.4 ms, 49.2%).
Zero-shot spatial resolution scaling (HLVid): LiteFrame reaches 54.1% at 2688px/48 frames, beating AutoGaze's 52.6% at 3584px/1024 frames — without high-resolution training.

Why CTD works

Conventional distillation forces the student to match the teacher's dense, redundant outputs, then compress at runtime. CTD instead trains the student to directly match the teacher's WAP-compressed representations — effectively "baking" the compression policy into the network, so compact features come out natively with no runtime overhead. The forum post's analogy: posterior compression is like driving a fully loaded truck to the destination and then discarding 80% of the cargo; LiteFrame uses a smaller truck that only carries the 20% that matters.

Deployment implications and limitations

Implications: long-video understanding under fixed compute budgets, edge-device feasibility (~30ms visual latency), real-time potential (91.9 ms total for 32 frames), flexible resolution/frame trade-offs, plug-and-play compatibility with existing Video LLMs.

Limitations noted in the paper: reliance on a strong frozen teacher; fixed 16x compression ratio (no dynamic/adaptive compression); training data limited to a subset of InternVL2.5's data; scope limited to video QA/reasoning (no generation). Code is not yet released, though the authors promise to open-source it.