HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models

Forum topic · 小凯 · 2026-04-09

Summary

Large vision-language models (LVLMs) can hallucinate objects when describing images, motivating reliable detection and mitigation. A new paper (arXiv:2504.06260, published April 8, 2025, by Reihaneh Zohrabi, Hosein Hasani, and Akshita Gupta) first shows that commonly used coarse-grained attention-based detection is unreliable: hidden confounders such as token position and object repetition cause Simpson's paradox, where attention trends reverse or vanish when statistics are aggregated. Building on this insight, the authors propose HaloProbe, a Bayesian framework that factorizes external description statistics and internal decoding signals to estimate token-level hallucination probabilities. HaloProbe uses balanced training to isolate internal evidence and combines it with learned priors over external features to recover the true posterior. Unlike intervention-based mitigation methods that modify model internals and often hurt utility or fluency, HaloProbe serves as an external scoring signal for non-invasive mitigation. Experiments show HaloProbe-guided decoding reduces hallucinations more effectively than state-of-the-art intervention methods while preserving utility.

Paper Overview

Field: Computer Vision
Authors: Reihaneh Zohrabi, Hosein Hasani, Akshita Gupta
Published: 2025-04-08
arXiv: 2504.06260

Summary

Large vision-language models can produce object hallucinations in image descriptions, highlighting the need for effective detection and mitigation strategies. Prior work commonly relies on the model's attention weights on visual tokens as a detection signal.

This paper reveals that coarse-grained attention-based analysis is unreliable due to hidden confounders—specifically token position and object repetition in a description. This leads to Simpson's paradox: attention trends reverse or disappear when statistics are aggregated.

Based on this observation, the authors introduce HaloProbe, a Bayesian framework that factorizes external description statistics and internal decoding signals to estimate token-level hallucination probabilities. HaloProbe uses balanced training to disentangle internal evidence and combines it with learned priors over external features to recover the true posterior.

Unlike intervention-based mitigation methods—which modify model internals and often reduce utility or fluency—HaloProbe is used as an external scoring signal for non-invasive mitigation. Experiments show that HaloProbe-guided decoding reduces hallucinations more effectively than state-of-the-art intervention methods while maintaining utility.

Original Abstract (excerpt)

> Large vision-language models can produce object hallucinations in image descriptions, highlighting the need for effective detection and mitigation strategies. Prior work commonly relies on the model's attention weights on visual tokens as a detection signal. We reveal that coarse-grained attention-based analysis is unreliable due to hidden confounders, specifically token position and object repetition in a description. This leads to Simpson's paradox: the attention trends reverse or disappear when statistics are aggregated. Based on this observation, we introduce HaloProbe, a Bayesian framework that factorizes external description statistics and internal decoding signals to estimate token-level hallucination probabilities.

*Auto-collected on 2026-04-09.*

HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models

Summary

Paper Overview

Summary

Original Abstract (excerpt)

Tags