TPIPS: A Text-Prompted Image Perceptual Similarity Metric (arXiv 2607.18237)
Forum topic · 小凯 · 2026-07-22
Summary
Human judgments of visual similarity are context-dependent: two images may be similar in shape yet very different in color. Existing perceptual similarity metrics compress these nuances into a single scalar value and cannot be conditioned on specific aspects. Researchers including Sheng-Yu Wang, Yotam Nitzan, Aaron Hertzmann, Jun-Yan Zhu, and Eli Shechtman introduce TPIPS (Text-Prompted Image Perceptual Similarity), presented in arXiv paper 2607.18237. The work contributes a large-scale dataset of human similarity judgments, collected on image triplets and annotated along multiple free-form semantic dimensions of similarity. Extensive benchmarking of state-of-the-art vision-language models (VLMs) reveals a substantial gap versus human annotator consensus. Fine-tuning VLMs on this data yields the TPIPS metric, which captures multiple senses of visual similarity specified by a text prompt. TPIPS aligns more closely with human perception and reliably generalizes beyond the training distribution. The authors demonstrate new capabilities in text-guided retrieval, compositional search, and fine-grained evaluation of generative models.
Paper Overview
- Field: Computer Vision (cs.CV, cs.LG)
- Authors: Sheng-Yu Wang, Yotam Nitzan, Aaron Hertzmann, Jun-Yan Zhu, Eli Shechtman, et al. (7 authors total)
- Published: 2026-07-20
- arXiv: 2607.18237
Motivation
Human visual similarity judgments are context-dependent. For example, two images may be similar in shape but very different in color. Existing perceptual similarity metrics collapse these nuances into a single scalar value and cannot be conditioned on specific aspects of similarity.Contribution
To bridge this gap, the authors introduce a large-scale dataset of human similarity judgments. Annotations are collected on image triplets, with each triplet rated along multiple free-form semantic dimensions of similarity.Findings
- Extensive benchmarking of state-of-the-art vision-language models (VLMs) shows a significant performance gap compared to the consensus of human annotators.
- Fine-tuning VLMs on the collected data produces TPIPS, a text-prompted image perceptual similarity metric that captures multiple senses of visual similarity conditioned on a specified text prompt.
- TPIPS aligns more closely with human perception and reliably generalizes beyond the training distribution.
Applications
The paper demonstrates new capabilities enabled by TPIPS in:
- Text-guided image retrieval
- Compositional search
- Fine-grained evaluation of generative models
Links
- Paper: https://arxiv.org/abs/2607.18237
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178446994