Capability ≠ Interpretability: 377 Human Judges Find Strongest Vision AI Features Hardest to Understand

Forum topic · 小凯 · 2026-05-21

Summary

A large-scale psychophysics study by Brown University, ELLIS Alicante, and imec (arXiv:2605.20337) measured the human interpretability of six vision Transformers with 377 participants and over 15,000 behavioral responses. Using two complementary tasks—localizability (predicting where a feature activates on novel images) and nameability (describing features in free text, scored via CLIP alignment)—the study found that every vision foundation model (DINOv2, DINOv3, CLIP, SigLIP) scored lower on human interpretability than supervised baselines (ViT-S, ViT-B), despite their superior downstream performance. Interpretability was statistically orthogonal to ImageNet classification, ADE20K segmentation, and perceptual grouping benchmarks. The strongest predictors of interpretability were feature activation locality (Spearman ρ up to 0.99) and coarse-grained semantic alignment with human category structure—not model size, accuracy, or attention alignment with human eye movements. Notably, DINOv2 elicited the highest participant confidence while ranking near-last in actual interpretability, revealing a dangerous 'interpretability illusion.' DINOv3's locally-optimized training suggests the gap is not an architectural inevitability. The results argue that interpretability should be treated and optimized as an independent dimension of model quality.

Capability ≠ Interpretability: Human Interpretability of Vision Foundation Models

Paper info

| Item | Details | |------|---------| | Title | Capability ≠ Interpretability: Human Interpretability of Vision Foundation Models | | Authors | Julien Colin (Brown U / ELLIS), Lore Goetschalckx (imec), Nuria Oliver (ELLIS), Thomas Serre (Brown U) | | Institutions | Brown University, ELLIS Alicante, imec | | arXiv | 2605.20337 | | Date | 2026-05-19 | | Category | cs.CV | | Core claim | Stronger vision foundation models are not more human-interpretable. Interpretability is an independent dimension, orthogonal to downstream performance. What predicts it is feature activation locality and coarse-grained semantic alignment—not model size or accuracy. |

The paradox

Vision foundation models crush supervised models on downstream benchmarks, yet the visual features they learn are *harder for humans to understand*. A large-scale psychophysical experiment (377 participants, 15,000+ behavioral responses) tested six mainstream vision Transformers and found that every foundation model scored below supervised baselines on human interpretability.

How interpretability was measured

Two complementary tests were used:

Localizability: Participants saw a feature's synthetic visualization, nine highly-activating natural images, and activation heatmaps—then clicked where they predicted the feature would activate on a *new* image. This tests spatial understanding, not just knowing what a feature "cares about."
Nameability: With the same materials, participants described the feature in free text. Descriptions were scored by semantic similarity to the feature's actual attended regions in CLIP's joint vision-language space.

The two scores were highly correlated (r = 0.84), suggesting they measure the same underlying property.

Method details

Models tested: two supervised baselines (ViT-S/16, ViT-B/16, ImageNet supervision) and four foundation models (DINOv2, DINOv3 — self-supervised; CLIP, SigLIP — vision-language pretraining).

Monosemantic features via sparse autoencoders (SAEs): Since ViT neurons are polysemantic, the authors used TopK SAEs on last-layer representations to recover monosemantic directions (10× expansion: 3,840 features for ViT-S; 7,680 for others).

Functionally aligned feature selection: Features were matched across models by which images most strongly activate them, ensuring all six feature sets target similar visual content.

Probability-anchored scoring: Because heatmap shapes differ (focused vs. diffuse), raw click-activation scores are biased. The normalization anchors "random clicks" at 0.5, making scores comparable across features and models.

Why old methods fail: A control showed forced-choice protocols let participants succeed by elimination—above-chance accuracy (53% for ViT-B32, 60% for ResNet-50) persisted even with irrelevant reference images. Correcting for model-specific baselines reverses previously published interpretability rankings (e.g., Zimmermann et al.'s ResNet-50 vs. ViT-B32 result).

Core results: capability ≠ interpretability

| Model | Localizability ↑ | Nameability ↑ | Confidence | |-------|------------------|---------------|------------| | ViT-S (supervised) | 79.7 | 0.274 | 3.43 | | ViT-B (supervised) | 74.0 | 0.273 | 3.68 | | DINOv2 | 71.4 | 0.259 | 3.68 | | DINOv3 | 73.3 | 0.260 | 3.41 | | CLIP | 74.7 | 0.266 | 3.51 | | SigLIP | 67.1 | 0.253 | 3.38 |

Both supervised models ranked first; differences are statistically significant (Kruskal-Wallis, p < 0.001).

Correlations with downstream performance were not significant for any benchmark:

| Benchmark | Localizability (ρ) | Nameability (ρ) | |-----------|--------------------|-----------------| | ImageNet classification | -0.48 (p=0.33) | -0.60 (p=0.21) | | ADE20K segmentation | -0.54 (p=0.27) | -0.77 (p=0.07) | | Perceptual grouping | 0.12 (p=0.82) | 0.09 (p=0.87) |

Interpretability is not a trade-off with capability—it is statistically orthogonal. Benchmarks that drive AI progress today say nothing about whether models are "transparent" to humans.

What actually predicts interpretability

1. Feature activation locality — the strongest predictor. Hoyer sparsity of feature activations correlates at ρ = 0.91 (p = 0.01) with localizability and ρ = 0.99 (p < 0.001) with nameability. Focused features are understandable; features blending objects with background context are not. 2. Coarse-grained semantic alignment — significant correlation with human category structure (localizability ρ = 0.84, p = 0.04; nameability ρ = 0.85, p = 0.03). Notably, alignment with human eye movements (ρ ≈ 0.14–0.21) and fine-grained perceptual similarity judgments did *not* predict interpretability. A model can match human gaze patterns perfectly yet encode incomprehensible features; what matters is whether its features reflect human coarse category organization.

The dangerous blind spot: DINOv2's "confidence illusion"

DINOv2 elicited the highest participant confidence (3.68/5) while ranking near-last in actual interpretability. Participants believed they understood its features—but failed when asked to predict behavior on novel images. Since DINOv2 is widely treated as a reference for interpretable representations (including in medical imaging and autonomous driving), this systematic overestimation poses real deployment risks.

DINOv3: the exception proving the rule

DINOv3 was the only foundation model approaching supervised models. Its training objective explicitly promotes local features—showing the interpretability gap is not an architectural inevitability and can be directly optimized without sacrificing performance.

Honest limitations

Causality unverified: Locality and coarse alignment predict interpretability, but no interventional experiments confirm causal effects.
Scale: Only ViT-Base/Small models were tested; dynamics at ViT-Large/Huge scale are unknown.
Modality: The framework is vision-only; extension to LLMs is undiscussed.
SAE optimality: Whether SAE-extracted features faithfully reflect the model's learned structure, and whether rankings are stable across SAE configurations, was not ablated.
Nameability noise: Free-text scoring introduces linguistic-ability variance and CLIP's own biases into the metric.

References

1. Colin, J., Goetschalckx, L., Oliver, N., & Serre, T. (2026). *Capability ≠ Interpretability: Human Interpretability of Vision Foundation Models.* arXiv:2605.20337 2. Zimmermann, R. S. et al. (2024). *Scale alone does not improve mechanistic interpretability in vision models.* NeurIPS 2024. 3. Bau, D. et al. (2017). *Network dissection: Quantifying interpretability of deep visual representations.* CVPR 2017. 4. Borowski, J. et al. (2021). *Exemplary natural images explain CNN activations better than state-of-the-art feature visualization.* ICLR 2021. 5. Fel, T. et al. (2023). *Unlocking feature visualization for deeper networks with magnitude constrained optimization.* NeurIPS 2023. 6. Cunningham, H. et al. (2023). *Sparse autoencoders find highly interpretable features in language models.* arXiv:2309.08600.