Extreme Compression of Visual Cortex Models: 5000x Parameter Reduction Reveals the Simplicity of Brain Computation

Paper Info

| Item | Details | |------|---------| | Title | Compact deep neural network models of visual cortex | | Authors | Benjamin R. Cowley, Patricia L. Stan, Jonathan W. Pillow, Matthew A. Smith | | Institutions | Cold Spring Harbor Laboratory, Princeton Neuroscience Institute, Carnegie Mellon University | | Published | bioRxiv, 2023.11.22.568315 | | DOI | 10.1101/2023.11.22.568315 | | Access | Open access (CC-BY-NC-ND 4.0) |

One-Sentence Summary

Using knowledge distillation and deep pruning, this study compressed neural prediction models of macaque V4 from ~90 million parameters down to ~10,000 (~5000x compression) while maintaining high prediction accuracy (R²=0.55)—outperforming ResNet50 and revealing that biological vision achieves diverse feature selectivity by "precisely merging" a small set of shared Gabor-like filters.

Background: The Black-Box Problem

Deep neural networks predict visual cortex responses remarkably well—ResNet50 intermediate activations can predict V4 and IT responses. But these models contain tens of millions of parameters and are nearly uninterpretable. Are we merely replacing one complex biological system with another complex artificial one? Classic linear-nonlinear models work for retina and V1 but fail for higher areas like V4, whose neurons show diverse selectivity for edges, curves, textures, and colors.

Core question: How small can a model be while still predicting V4 responses?

Experimental Design

• Animals: 3 male rhesus macaques, 96-channel Blackrock arrays over left V4 (prelunate gyrus)
• Data: 50 sessions, ~2,600 neurons total, 219 held-out test neurons, ~78,000 unique images (~2,000/session, 2–24 repeats), 100 ms response window from 50 ms post-stimulus
• Closed-loop active learning: images where the ensemble disagreed most were preferentially shown; additionally, high-contrast binarized "gaudy" images proved even more efficient for training than active learning

Three-Stage Model Pipeline

Stage 1: Deep Ensemble (~90M parameters)

Frozen ResNet50 (activation_33 layer) feeds 25 trainable ensemble DNNs with 1x1 convolutions, separable convolutions, and 4 ResNet-like skip-connection blocks, ending in a factorized linear readout that reduces parameters from p²k to p² + k, mitigating overfitting.

Stage 2: Knowledge Distillation (~600 filters)

A small 5-layer CNN (100 filters per layer, 5x5 kernels) was trained on the ensemble's predictions for 12 million images. Directly training the small network on real data gave only R²=0.11 due to overfitting; distillation leverages the ensemble's abundant soft labels.

Stage 3: Depth-wise Channel Pruning (~150 filters)

Filters were ranked by output variance, removed starting from the deepest layer (keeping 90% of each layer's variance), retrained, and pruned until accuracy dropped >5%. Result: a unique compact model (~10,000 parameters) per V4 neuron—219 in total.

Performance: Small Is Beautiful

| Model | Median noise-corrected R² | |-------|---------------------------| | Untrained ResNet50 | 0.13 | | ResNet50 | 0.45 | | Compact models | 0.55 | | Deep ensemble | 0.60 |

• Compression ratio: ~5000x vs. the ensemble (up to 9000x); ~500x vs. task-driven DNNs
• Noise-corrected R² (Pospisil & Bair, 2021) is an unbiased estimator; conventional R² overestimates by ~0.15

Causal Validation

Probe images—maximizing natural images, gradient-ascent synthesized images, and adversarial perturbations—were shown in later sessions. Real V4 responses matched compact-model predictions, confirming the models capture genuine stimulus preferences.

Key Findings: Shared Basis + Precise Merging

1. Early-layer filters are highly shared: kernel similarity across the 219 compact models is near 1 in layers 1–2 (Gabor-like), moderate in layer 3, low in layers 4–5. 2. The merge step drives specialization: CKA analysis shows early representations are similar while later/output representations diverge (output-layer mean signal correlation squared ρ² = 0.11). V4 diversity comes from *how shared features are merged*, not from independent early pathways. 3. Very few shared filters suffice: a shared compact model with 50 filters per early layer (across all neurons) matches the full models; 10 per layer already beats ResNet50. Random frozen early layers perform significantly worse—filters must be trained.

Case Study: Dissecting a Dot Detector

For one compact model preferring small dots, ablation analysis (Dot Size Invariance metric) found:

• No single layer-1–3 filter was a strong contributor, yet ~10 layer-3 filters jointly carry the selectivity
• Mechanism: 4 excitatory filters respond to corner-like curves of small dots with spatially overlapping outputs; 2 suppressive filters respond to long edges (present in large dots only). Sum + ReLU yields strong activity for small dots, weak activity for large dots—a classic excitatory-convergence-plus-suppression circuit

Why the Compression Framework Works

1. Lottery Ticket Hypothesis: large networks contain sparse trainable subnetworks that pruning can find 2. Distillation solves data scarcity: 12M synthetic predictions substitute for ~2,000 real images per session 3. Channel-wise pruning keeps kernels smooth and interpretable 4. Pruning deepest-first preserves early-layer information

Discussion

• Task-driven DNNs like ResNet50 appear "unnecessarily large" for neural prediction—a smaller specialized model is both more accurate and more interpretable
• V1/V2 neurons are likely reused across multiple V4 neurons' specializations, consistent with broad, overlapping V1→V4 projections
• Testable predictions: anatomical tracing of divergent V1/V2→V4 projections; circuit perturbations of specific filter types; even smaller models with surround suppression, divisive normalization, or recurrence
• Limitations: feature encoding only; static images; single area; simple convolutional architecture

References

1. Cowley BR, Stan PL, Pillow JW, Smith MA. *Compact deep neural network models of visual cortex*. bioRxiv. 2023;568315. DOI: 10.1101/2023.11.22.568315 2. Yamins DLK, DiCarlo JJ. Using goal-driven deep learning models to understand sensory cortex. *Nature Neuroscience*. 2016;19(3):356-365. 3. Pospisil DA, Bair W. The unbiased estimation of the fraction of variance explained by a model. *PLoS Computational Biology*. 2021;17(8):e1009212. 4. Frankle J, Carbin M. The lottery ticket hypothesis. *ICLR*. 2019. 5. Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network. *NIPS Deep Learning Workshop*. 2014.