HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models

Forum topic · 小凯 · 2026-04-09

Summary

Large vision-language models (LVLMs) can hallucinate objects in image descriptions, and prior detection methods often rely on the model's attention weights on visual tokens. This arXiv paper (2504.06260) by Reihaneh Zohrabi, Hosein Hasani, and Akshita Gupta shows that coarse-grained attention-based analysis is unreliable because of hidden confounders such as token position and object repetition, producing Simpson's paradox where attention trends reverse or vanish when statistics are aggregated. The authors propose HaloProbe, a Bayesian framework that factorizes external description statistics and internal decoding signals to estimate token-level hallucination probabilities. HaloProbe uses balanced training to isolate internal evidence and combines it with a learned prior over external features to recover the true posterior. Unlike interventional mitigation approaches that modify model internals and can degrade utility or fluency, HaloProbe serves as an external scoring signal for non-invasive mitigation. Experiments show that HaloProbe-guided decoding reduces hallucinations more effectively than state-of-the-art interventional methods while preserving model utility.

Overview

Field: Computer Vision (CV)
Authors: Reihaneh Zohrabi, Hosein Hasani, Akshita Gupta
Published: 2025-04-08
arXiv: 2504.06260

Summary

Large vision-language models can produce object hallucinations in image descriptions, highlighting the need for effective detection and mitigation strategies. Prior work commonly relies on the model's attention weights on visual tokens as a detection signal. The paper reveals that coarse-grained attention-based analysis is unreliable due to hidden confounders—specifically token position and object repetition in a description. This leads to Simpson's paradox: attention trends reverse or disappear when statistics are aggregated.

Based on this observation, the authors introduce HaloProbe, a Bayesian framework that factorizes external description statistics and internal decoding signals to estimate token-level hallucination probabilities. HaloProbe employs balanced training to disentangle internal evidence and combines it with a learned prior over external features to recover the true posterior.

Unlike interventional mitigation methods that modify model internals and can reduce utility or fluency, HaloProbe is used as an external scoring signal for non-invasive mitigation. Experiments demonstrate that HaloProbe-guided decoding reduces hallucinations more effectively than state-of-the-art interventional approaches while maintaining utility.

Abstract (original)

> Large vision-language models can produce object hallucinations in image descriptions, highlighting the need for effective detection and mitigation strategies. Prior work commonly relies on the model's attention weights on visual tokens as a detection signal. We reveal that coarse-grained attention-based analysis is unreliable due to hidden confounders, specifically token position and object repetition in a description. This leads to Simpson's paradox: the attention trends reverse or disappear when statistics are aggregated. Based on this observation, we introduce HaloProbe, a Bayesian framework that factorizes external description statistics and internal decoding signals to estimate token-level hallucination probabilities.

HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models

Summary

Overview

Summary

Abstract (original)

Tags