LiteFrame: 71% Smaller Video Encoder Unlocks 8× Frame Scaling in Video LLMs

> Source: zhichai.net forum post. Paper: "LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs" — Google DeepMind & Seoul National University, arXiv: 2605.17260. Project page: https://jjihwan.github.io/projects/LiteFrame

The blind spot in Video LLM efficiency

Modern Video LLMs run a three-stage pipeline: a ViT encoder extracts per-frame features (~256 tokens/frame), a projector aligns them to the LLM space, and the LLM reasons over interleaved visual-text tokens. Most efficiency work (ToMe, PruMerge, FastVID) compresses visual tokens after a 304M-parameter ViT-Large runs per frame — so when the LLM speeds up, encoder latency simply becomes the new dominant bottleneck. Post-hoc compression alone yields little end-to-end improvement on long videos.

What LiteFrame does

LiteFrame internalizes token compression into the vision backbone:

• Lightweight student encoder: ViT-Base-12L (87M params, hidden dim 768) vs. the 304M ViT-Large-24L teacher (-71% parameters).
• DWTempConv: spatial attention layers interleaved with depthwise separable 1D temporal convolutions, with strided temporal downsampling inserted after layers 4 and 8. On 256 frames, DWTempConv achieves 174.84 ms latency and 17.92 TFLOPs vs. the teacher's 1043.33 ms / 158.80 TFLOPs.
• WAP (Weighted Average Pooling): an attention-weighted spatiotemporal pooling operator (16× compression) that outperforms average/max pooling, subsampling, ToMe, PruMerge, and FastVID as a post-hoc method — and also serves as the supervision target generator for distillation.
• CTD (Compressed Token Distillation): instead of aligning with the teacher's raw dense outputs, the student is trained on L_CTD(θ) = ||S_θ(x) - P(T(x))||², where P(·) is WAP applied to the frozen teacher's features. This bakes the compression policy into the student's weights, so no runtime pruning is needed.
• LMA: LoRA-based LLM adaptation (r=4, α=8) with frame sampling of {128, 256, 512} at 1–4 FPS.

Headline results (vs. InternVL3-8B baseline)

| Baseline | Method | Frames | Visual params | Total latency | Avg accuracy | |---|---|---|---|---|---| | 8 frames | InternVL3-8B | 8 | 304M | 208.4 ms | 59.0% | | | +FastVID | 32 (4×) | 304M | 224.8 ms (+7.9%) | 59.5% | | | +LiteFrame | 64 (8×) | 87M | 150.1 ms (-28.0%) | 61.1% (+2.1) | | 16 frames | +LiteFrame | 128 (8×) | 87M | 272.6 ms (-32.5%) | 63.7% (+1.5) | | 32 frames | +LiteFrame | 256 (8×) | 87M | 532.3 ms (-34.6%) | 65.7% (+0.4) |

Additional findings:

• 12.7× faster LLM prefill at 64 frames; 5.3× faster ViT encoding.
• Redefines the latency-accuracy Pareto frontier, beating ToMe/FastVID/PruMerge at 16× compression, and outperforming AutoGaze (whose total latency grows +682% when scaling frames vs. LiteFrame's -34.6%).
• Zero-shot spatial resolution scaling: 2688px / 48 frames reaches 54.1% on the HLVid benchmark, beating AutoGaze (3584px / 1024 frames, 52.6%) without high-resolution training.
• Against other efficient encoders at 32 frames: 91.9 ms total latency and 58.0% accuracy vs. FastVLM (231.5 ms, 47.6%) and VideoPanda (382.4 ms, 49.2%).

Why CTD works

Conventional distillation forces the student to reproduce the teacher's redundant dense features, which the system then compresses anyway at runtime. CTD instead aligns the student with the teacher's WAP-compressed, information-dense representations — the attention-weighting "selection criteria" are fixed into the student's parameters during training, yielding compact outputs natively at inference.

Limitations noted in the post

• Trained on a subset of InternVL2.5 data; larger/more diverse data unexplored.
• Requires a strong frozen teacher; student quality is teacher-bounded.
• Fixed 16× compression ratio; no dynamic/adaptive compression.
• Focused on video QA/reasoning, not generation or interactive multimodality.

Takeaway

The post's core argument: end-to-end system latency, not LLM speed, is what matters. Post-hoc token compression is like trucking a full load to its destination and discarding 80% of the cargo — LiteFrame instead uses a smaller truck trained to carry only what matters, achieving frame scaling and latency reduction simultaneously with no accuracy loss.