Stop the Inflate-Then-Deflate Waste: Compressed-Domain Deep Learning on JPEG and Video Streams

Forum topic · 小凯 · 2026-08-26

Summary

This post argues that most computer vision systems waste compute by 'inflating' compact compressed data back into redundant pixels, only for the first convolution layers to re-learn frequency features. Grounded in compressed sensing, it explains that natural images are already sparse in the DCT basis, so neural networks can reason directly on compressed-domain inputs. Four landmark papers are reviewed: Gueguen et al. (NeurIPS 2018) feed 28x28x192 DCT tensors straight from JPEG, boosting ResNet-50 throughput 1.77-2x with unchanged accuracy; Xu et al. (CVPR 2020 Oral) prune 87.5% of high-frequency channels and gain +1.4% Top-1 on ImageNet; CoViAR (CVPR 2018) performs action recognition directly on I-frames, motion vectors, and residuals, reaching 400+ FPS (10-50x faster than optical-flow pipelines); F3-Net (ECCV 2020) detects face forgeries via high-frequency DCT artifacts with 93.8-98.8% AUC on FaceForensics++. Reported gains include 60-80% compute reduction and 2-50x speedups.

The Problem: Inflate, Then Deflate

The post opens with an analogy: a factory ships a neatly folded umbrella (a JPEG file), the receiver fully unfolds it into a huge umbrella (CPU runs IDCT decoding into massive RGB pixels), trucks the bulk over full bandwidth, and then the GPU's first convolution layer laboriously re-folds it to recover edges and textures. Many vision systems repeat this wasteful cycle daily.

Compressed sensing shows that natural images and video, once projected onto bases like the DCT (plus motion compensation), are already highly sparse ( $K \ll N$ ). Recent research from NeurIPS, CVPR, and ECCV demonstrates that skipping the decode/inverse-transform and running inference directly in the compressed domain (DCT coefficients, motion vectors) can cut computation and data transfer by 60-80%, speed up inference 2-50x, and even *improve* accuracy on classification and fake-face detection.

Information Redundancy and the 'Reverse Inflation'

Adjacent pixels in natural images correlate at $\rho \approx 0.95$ , so the spatial pixel domain is highly redundant.
JPEG maps $8 \times 8$ blocks onto the DCT basis: $\mathbf{x}_{\text{pixel}} = \mathbf{\Psi}_{\text{DCT}} \boldsymbol{\theta}_{\text{freq}}$ , with energy concentrated in DC and a few low-frequency AC coefficients — a natural sparsity prior.

Traditional pipeline (reverse inflation): compact DCT coefficients → forced IDCT → bloated pixels → Conv1 re-learns frequency features.

Compressed-domain learning: compact DCT coefficients → straight into the network → fast semantic output.

> Compressed-Domain Feature Learning: a technique that skips decoding (IDCT, dequantization, full pixel reconstruction) and feeds frequency-domain coefficients, motion vectors, and residuals directly into neural networks as input tensors.

Four Landmark Papers

1. Faster Neural Networks Straight from JPEG (NeurIPS 2018, Uber AI)

Skips CPU-side IDCT and YCbCr→RGB conversion; organizes each 8x8 DCT block into channel dimensions, turning a $224 \times 224 \times 3$ image into a compact $28 \times 28 \times 192$ tensor (64x smaller spatially).
Results: ResNet-50 throughput up 1.77-2.0x on ImageNet, first-layer compute down 60%, Top-1 accuracy identical (76.2% vs 76.1%).

2. Learning in the Frequency Domain (CVPR 2020 Oral; Alibaba DAMO Academy & ASU)

A dynamic frequency-channel selection network evaluates the semantic contribution of 192 DCT bands; most quantized high-frequency AC channels are mostly noise.
Pruning 87.5% of high-frequency channels (keeping 24 channels, $28 \times 28 \times 24$ ) *raised* ImageNet Top-1 by +1.4% on ResNet-50 and generalized well to COCO detection and Mask R-CNN segmentation.

3. CoViAR: Compressed Video Action Recognition (CVPR 2018; UT Austin, CMU, AWS)

Decoding H.264/H.265 streams to RGB frames consumes over 80% of server CPU time in traditional pipelines.
CoViAR directly uses three stream structures: I-frame DCT texture, P-frame motion vectors (free optical flow), and residuals, fed into a three-stream CNN.
Results: 10-50x faster than optical-flow two-stream networks; 400+ FPS on a single consumer GPU on UCF-101.

4. F3-Net: Frequency-Aware Face Forgery Detection (ECCV 2020; SenseTime & Beihang University)

GAN/diffusion-generated faces are spatially smoothed, hiding artifacts from eyes and standard CNNs — but periodic upsampling/deconvolution grid artifacts leave visible periodic spikes in the high-frequency DCT spectrum.
Results: 93.8-98.8% AUC on FaceForensics++ (the source cites 98.8%), validating frequency representations against stealthy forgeries.

Why Frequency-Domain Input Is Efficient

Conclusion

Optimization is not limited to stacking more parameters or layers. Respecting the inherent structure of information — leveraging sparsity priors and computing directly on compact orthogonal bases — can deliver multiplicative efficiency gains at very low cost. From static-image DCT inference to motion-vector-driven video analytics, frequency-domain deep learning is becoming a key enabler for edge computing and large-scale real-time intelligence.

References

1. Gueguen, L., Sergeev, A., Kadlec, B., Liu, R., & Yosinski, J. (2018). *Faster Neural Networks Straight from JPEG*. NeurIPS 2018, 31, 3933-3944. 2. Xu, K., Qin, M., Sun, F., Wang, Y., Chen, Y. K., & Ren, F. (2020). *Learning in the Frequency Domain*. CVPR 2020 (Oral), 1740-1749. 3. Wu, C. Y., Zaheer, M., Hu, H., Manmatha, R., Smola, A. J., & Krähenbühl, P. (2018). *Compressed Video Action Recognition*. CVPR 2018, 6026-6035. 4. Qian, Y., Yin, G., Sheng, L., Chen, Z., & Shao, J. (2020). *Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues*. ECCV 2020, 86-103. 5. Candès, E. J., & Wakin, M. B. (2008). *An introduction to compressive sampling*. IEEE Signal Processing Magazine, 25(2), 21-30.