Capability ≠ Interpretability: 377 Human Raters Find the Strongest Vision AI Models Are the Hardest to Understand

Forum topic · 小凯 · 2026-05-21

Summary

A large-scale psychophysics study by Brown University, ELLIS Alicante, and imec (arXiv:2605.20337) measured how interpretable vision foundation model features are to humans. 377 participants completed over 15,000 behavioral trials across two tasks—localizing where a feature activates on new images, and freely naming what a feature represents—for six Vision Transformer models: supervised ViT-S and ViT-B baselines plus DINOv2, DINOv3, CLIP, and SigLIP. Using sparse autoencoders to extract monosemantic features and a probability-anchored scoring function, the authors found that every foundation model scored below the supervised baselines, and interpretability showed no significant correlation with downstream performance on ImageNet classification, ADE20K segmentation, or perceptual grouping. The strongest predictors of interpretability were feature activation locality (rho up to 0.99) and coarse-grained semantic alignment with human category structure. Notably, DINOv2 induced the highest participant confidence despite ranking near the bottom, revealing a dangerous 'interpretability illusion,' while DINOv3 nearly closed the gap by explicitly encouraging local features. Interpretability, the authors argue, is an axis orthogonal to capability.

Paper Overview

Title: Capability ≠ Interpretability: Human Interpretability of Vision Foundation Models
Authors: Julien Colin (Brown University / ELLIS), Lore Goetschalckx (imec), Nuria Oliver (ELLIS), Thomas Serre (Brown University)
arXiv: 2605.20337
Field: Computer Vision (cs.CV)
Core claim: A vision model's capability does not imply that its features are human-interpretable. Interpretability is an independent dimension, predicted not by model size or accuracy but by feature activation locality and coarse-grained semantic alignment.

The Core Paradox

Vision foundation models crush older supervised models on downstream tasks—yet their learned features are *harder* for humans to understand. Like a guide who can navigate the forest perfectly but cannot describe a pine tree, the strongest models may be the least transparent. This paper quantifies that paradox with a large-scale behavioral experiment.

How They Measured Interpretability

Two complementary tests, both requiring genuine understanding of a feature:

1. Localizability: After seeing a feature's visualization, top-activating natural images, and activation heatmaps, participants clicked on a *new* image where they predicted the feature would activate. A spatial prediction test. 2. Nameability: Participants freely described what the feature represents in text. Descriptions were compared to the feature's actual semantic content via the CLIP vision-language space.

The two scores were highly correlated (r = 0.84), suggesting they measure the same underlying property.

Methodology

Models tested: supervised ViT-S/16 and ViT-B/16 (ImageNet supervision) vs. foundation models DINOv2, DINOv3 (self-supervised), CLIP, SigLIP (language-image pretraining).
Sparse autoencoders (TopK SAEs) recovered monosemantic feature directions from the final layer (10× overcomplete; 3,840 features for ViT-S, 7,680 for others), avoiding the polysemanticity of individual neurons.
Functionally aligned feature selection: features across models were matched by which images most activate them, ensuring comparable input-level concepts.
Probability-anchored scoring: random clicks were anchored to 0.5, with clicks above a feature's mean activation scoring positive—making scores comparable across features with very different heatmap shapes.
Why old methods fail: In a forced-choice control, participants could guess correctly by elimination even when shown *irrelevant* activation images (53% for ViT-B32, 60% for ResNet-50 vs. 50% chance). After correcting for model-specific baselines, previously reported interpretability rankings (e.g., Zimmermann et al.'s ResNet-50 > ViT-B32) reverse.

Key Results

| Model | Localizability ↑ | Nameability ↑ | |-------|------------------|----------------| | ViT-S (supervised) | 79.7 | 0.274 | | ViT-B (supervised) | 74.0 | 0.273 | | DINOv2 | 71.4 | 0.259 | | DINOv3 | 73.3 | 0.260 | | CLIP | 74.7 | 0.266 | | SigLIP | 67.1 | 0.253 |

Both supervised models outperformed all four foundation models (Kruskal-Wallis, p < 0.001).
No significant correlation with downstream performance: ImageNet classification (ρ = −0.48, p = 0.33 for localization), ADE20K segmentation (ρ = −0.54, p = 0.27), perceptual grouping (ρ = 0.12, p = 0.82). Interpretability is statistically orthogonal to capability—not a trade-off, not a free lunch.

What Actually Predicts Interpretability

1. Feature activation locality — the strongest predictor. Model-level Hoyer sparsity correlates with nameability at ρ = 0.99 (p < 0.001) and localizability at ρ = 0.91 (p = 0.01). Features focused on small, well-defined regions are understandable; diffuse activations blending objects and context are not. 2. Coarse-grained semantic alignment — models whose feature space matches human category structure ("is a buffalo more like grass than a spider is?") are more interpretable. Fine-grained perceptual similarity (butterfly-level distinctions) and even eye-movement alignment showed no relationship.

The Dangerous Blind Spot: DINOv2's Confident Illusion

DINOv2 produced the highest participant confidence (3.68/5) while ranking near the bottom in actual interpretability. Participants *felt* they understood its features but failed at predicting their behavior. Given DINOv2's adoption as a reference backbone in medical imaging and autonomous driving, this systematic overestimation poses real deployment risks.

DINOv3, by contrast, nearly closed the gap—because its training objective explicitly promotes local features. The interpretability gap is not an architectural fate: it can be optimized directly without sacrificing downstream performance.

Honest Limitations

No causal validation: locality and alignment predict interpretability, but no interventional experiments confirm causation.
Scale: all models were ViT-S/Base; scaling dynamics at ViT-L/H remain untested.
Modality: the framework is vision-only; extension to LLMs is undiscussed.
SAE dependence: results assume the sparse autoencoder faithfully extracts the model's feature structure; SAE configuration choices were not ablated.
Nameability confounds: verbal ability varies across participants, and CLIP's own alignment biases may leak into the nameability score.

Takeaway

The metrics driving AI progress today—accuracy on benchmarks—are blind to whether models are transparent to humans. Interpretability must be measured, and potentially optimized, as its own dimension.

References

1. Colin, J., Goetschalckx, L., Oliver, N., & Serre, T. (2026). *Capability ≠ Interpretability: Human Interpretability of Vision Foundation Models.* arXiv:2605.20337. 2. Zimmermann, R. S. et al. (2024). *Scale alone does not improve mechanistic interpretability in vision models.* NeurIPS 2024. 3. Bau, D. et al. (2017). *Network dissection: Quantifying interpretability of deep visual representations.* CVPR 2017. 4. Borowski, J. et al. (2021). *Exemplary natural images explain CNN activations better than state-of-the-art feature visualization.* ICLR 2021. 5. Fel, T. et al. (2023). *Unlocking feature visualization for deeper networks with magnitude constrained optimization.* NeurIPS 2023. 6. Cunningham, H. et al. (2023). *Sparse autoencoders find highly interpretable features in language models.* arXiv:2309.08600.