[论文] HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations ...

## 论文概要 **研究领域**: CV **作者**: Reihaneh Zohrabi, Hosein Hasani, Akshita Gupta **发布时间**: 2025-04-08 **arXiv**: [2504.06260](https://arxiv.org/abs/2504.06260) ## 中文摘要 大型视觉语言模型在图像描述中可能产生对象幻觉，凸显了有效检测和缓解策略的必要性。先前工作通常依赖模型对视觉token的注意力权重作为检测信号。本文揭示，由于隐藏的混杂因素——特别是描述中的token位置和对象重复——粗粒度的基于注意力的分析是不可靠的。这导致了辛普森悖论：当统计聚合时，注意力趋势会逆转或消失。基于这一观察，我们引入HaloProbe，一个贝叶斯框架，将外部描述统计与内部解码信号分解以估计token级幻觉概率。HaloProbe使用平衡训练来分离内部证据，并将其与外部特征的学习先验结合以恢复真实的后验。与通常通过修改模型内部来降低效用或流畅性的干预式缓解方法不同，我们使用HaloProbe作为外部评分信号进行非侵入式缓解。实验表明，HaloProbe引导的解码比最先进的干预式方法更有效地减少幻觉，同时保持效用。 ## 原文摘要 Large vision-language models can produce object hallucinations in image descriptions, highlighting the need for effective detection and mitigation strategies. Prior work commonly relies on the model's attention weights on visual tokens as a detection signal. We reveal that coarse-grained attention-based analysis is unreliable due to hidden confounders, specifically token position and object repetition in a description. This leads to Simpson's paradox: the attention trends reverse or disappear when statistics are aggregated. Based on this observation, we introduce HaloProbe, a Bayesian framework that factorizes external description statistics and internal decoding signals to estimate token-level hallucination probabilities. --- *自动采集于 2026-04-09* #论文 #arXiv #CV #小凯