Compact DNN Models of Visual Cortex: 5000x Parameter Reduction Reveals Simple Computation in V4
This post summarizes the paper "Compact deep neural network models of visual cortex" by Benjamin R. Cowley, Patricia L. Stan, Jonathan W. Pillow, and Matthew A. Smith (Cold Spring Harbor Laboratory, Princeton Neuroscience Institute, Carnegie Mellon University), published on bioRxiv: 10.1101/2023.11.22.568315 (CC-BY-NC-ND 4.0).
Key points
- Extreme compression with better accuracy: Using knowledge distillation plus depth pruning, V4 prediction models were compressed from ~90M parameters (deep ensemble on ResNet50 features) to ~10,000 parameters (~150 filters, up to 9000x), reaching median noise-corrected R2 = 0.55 vs 0.45 for ResNet50 and 0.60 for the full ensemble.
- Data: 3 male macaques, 96-channel Utah arrays in V4, 50 sessions, ~2,600 neurons (219 held-out test neurons), ~78,000 unique images, with closed-loop active learning and high-contrast binarized "gaudy" images for efficient training.
- Causal validation: Maximizing natural images, gradient-ascent synthetic images, and adversarial images presented in later sessions confirmed that compact models match the real V4 neurons' stimulus preferences.
- Shared basis + precise merging: Early-layer filters are highly shared across all 219 per-neuron models; late layers and the output merge step drive each neuron's specialization (output-layer average squared signal correlation only ρ² = 0.11). A shared compact model with just 50 filters per layer explains all 219 V4 neurons, beating ResNet50 even with 10 filters per layer.
- Dot-detector case study: Ablation analysis shows size selectivity is not in any single early filter (layer-3 DSI < 0.5 individually) but emerges from the merge: ~10 layer-3 filters combine excitatory filters whose responses spatially overlap for small dots with inhibitory edge filters that suppress large edges.
- Implication: Task-driven DNNs like ResNet50 are "unnecessarily large" for neural prediction—biological vision achieves diverse selectivity by reusing a small set of low-level features (akin to V1 Gabor filters) through precise recombination.
Methods in brief
1. Stage 1 — Deep ensemble (~90M params): frozen ResNet50 features (112×112×3 input, activation_33 layer) → 25 trainable ensemble DNNs with separable convolutions and 4 residual blocks → factorized linear readout (spatial pooling × channel mixing, reducing parameters from p²k to p²+k) averaged over members. 2. Stage 2 — Distillation (~600 filters): a 5-layer compact CNN trained on the ensemble's predictions for 12M images (soft labels), solving data scarcity (direct training on ~2,000 real images per session gave R2 = 0.11). 3. Stage 3 — Depth pruning (~150 filters): channel-level pruning ranked by filter activity variance, deepest-first, keeping the minimal set explaining 90% of each layer's variance; retraining after pruning; stopping at >5% performance drop.
Evaluation
The paper uses noise-corrected R2 (Pospisil & Bair 2021), an unbiased estimator that lowers scores by ~0.15 relative to standard R2—suggesting previous literature may overstate model accuracy.
Testable predictions
1. Anatomical tracing should show V1/V2 neurons projecting broadly to multiple V4 neurons. 2. Perturbing specific V1 filter types should co-modulate multiple V4 neurons sharing those filters. 3. Adding biological mechanisms (surround suppression, divisive normalization, recurrence) may yield even smaller models.
Limitations
The work covers feature encoding of static images in one area; extensions include trial-to-trial variability, attention, dynamic vision, and other regions such as IT.
References
1. Cowley BR, Stan PL, Pillow JW, Smith MA. *Compact deep neural network models of visual cortex*. bioRxiv. 2023;568315. DOI: 10.1101/2023.11.22.568315 2. Yamins DLK, DiCarlo JJ. *Nature Neuroscience*. 2016;19(3):356-365. 3. Pospisil DA, Bair W. *PLoS Computational Biology*. 2021;17(8):e1009212. 4. Frankle J, Carbin M. *ICLR*. 2019. 5. Hinton G, Vinyals O, Dean J. *NIPS Deep Learning Workshop*. 2014.