Paper Overview
Field: Computer Vision (CV) Authors: Sheng-Yu Wang, Yotam Nitzan, Aaron Hertzmann, Jun-Yan Zhu, Eli Shechtman, et al. (7 authors total) Published: 2026-07-20 arXiv: 2607.18237 Categories: cs.CV, cs.LGSummary
Human judgments of visual similarity are context-dependent. For example, two images may be similar in shape but differ dramatically in color. However, existing perceptual similarity metrics compress these nuances into a single scalar value and cannot be conditioned on specific aspects of similarity.To bridge this gap, the authors introduce a large-scale dataset of human similarity judgments, collected on image triplets, where each triplet is annotated along multiple free-form semantic similarity dimensions.
Extensive benchmarking of state-of-the-art vision-language models (VLMs) reveals a significant performance gap compared to human annotator consensus. Leveraging this data, the authors fine-tune VLMs to produce TPIPS (Text-Prompted Image Perceptual Similarity), a metric that captures multiple semantic senses of visual similarity conditioned on a specified text prompt.
Key Contributions
- A large-scale human similarity judgment dataset built on image triplets with free-form, multi-dimensional annotations.
- Benchmarking that exposes a substantial gap between frontier VLMs and human consensus on similarity judgments.
- TPIPS, a fine-tuned, text-prompted perceptual similarity metric that aligns more closely with human perception and generalizes reliably beyond the training distribution.
- Demonstrated new capabilities in text-guided retrieval, compositional search, and fine-grained evaluation of generative models.