[论文] See & Sniff: Learning Visuo-Olfactory Representations

论文概要

研究领域: CV 作者: Seongyu Kim, Seungwoo Lee, Hyeonggon Ryu, Joon Son Chung, Arda Senocak 发布时间: 2026-06-25 arXiv: 2606.27307

中文摘要

虽然现代多模态模型将视觉与语言、音频或触觉整合，但由于缺乏配对的视觉-嗅觉数据，嗅觉在很大程度上未被探索。我们引入 SmellNet-V，一个可扩展的视觉-嗅觉数据集，基于以下洞察：气味身份在语义类别内的视觉变换下基本不变。这使我们能够将仅嗅觉样本与语义对齐的野外网络图像进行合成配对，将单模态嗅觉数据集转化为跨模态基准，而无需昂贵的协同采集。基于该数据集，我们提出 See & Sniff，一种自监督框架，通过密集局部对齐学习联合视觉-嗅觉表征，并自然生成气味显著性图以实现气味源的空间定位。我们进一步引入像素级气味定位任务及评估基准。我们的方法在仅嗅觉分类上超越纯嗅觉基线 7%，并泛化到跨模态检索和气味定位，将视觉-嗅觉学习确立为多模态感知的新方向。

原文摘要

While modern multimodal models integrate vision with language, audio, or touch, olfaction remains largely unexplored due to the lack of paired visuo-olfactory data. We introduce SmellNet-V, a scalable visuo-olfactory dataset built on the insight that odor identity is largely invariant to visual transformations within a semantic category. This allows us to synthetically pair smell-only samples with semantically aligned in-the-wild web images, converting a unimodal olfactory dataset into a cross-modal benchmark without costly co-collection. Building on this dataset, we propose See & Sniff, a self-supervised framework that learns joint visuo-olfactory representations via dense local alignment and naturally produces smell saliency maps for spatial grounding of odor sources. We further introduc...

--- *自动采集于 2026-06-28*

#论文 #arXiv #CV #小凯