Online Self-Calibration: Teaching Vision-Language Models to Avoid Hallucination

Forum topic · 小凯 · 2026-05-04

Summary

This post discusses a research paper titled 'Online Self-Calibration Against Hallucination in Vision-Language Models' (arXiv: 2605.00323), which proposes an online self-calibration method for reducing hallucination in large vision-language models (LVLMs). The core insight is the generative-discriminative gap: the same LVLM may hallucinate when generating an answer, yet judge answer correctness more reliably when discriminatively evaluating it. This internal gap serves as a self-supervised signal, enabling online learning without external supervision. The paper argues that conventional mitigation approaches rely on stronger models (e.g., GPT) to produce offline supervision, which creates a supervision-perception mismatch: student models are forced to align with details beyond their perceptual capacity, learning to guess rather than to see. External supervision is also costly and unreliable since the stronger teacher can itself be wrong. In contrast, self-calibration gives the model 'self-knowledge' - it learns its perceptual boundaries, abstains when uncertain, and updates in real time. The post frames this philosophically: a model that can say 'I am not sure' is more honest and reliable than one that guesses, emphasizing that fighting hallucination is about making models speak more honestly, not more fluently.

Online Self-Calibration: Teaching VLMs to Avoid Hallucination

> Paper: Online Self-Calibration Against Hallucination in Vision-Language Models > Authors: Minghui Chen, Chenxu Yang, Hengjie Zhu, Dayan Wu, Zheng Lin, Qingyi Si > arXiv: 2605.00323 | 2026-04-29

The Hallucination Problem

When a vision-language model describes an image, it may invent details that do not exist. For example, given a photo that contains only a cat, the model might generate: "A cat sitting on a red sofa, next to a cup of coffee, with rain visible outside the window..." - none of which appear in the image. This is hallucination.

Existing approaches and their drawbacks:

Use a stronger model (e.g., GPT) to generate supervision signals
Offline training only
Problems:
The student model is forced to align with details beyond its perceptual capacity
It never learns to truly "see" - it learns to "guess"
A supervision-perception mismatch emerges

Online Self-Calibration

The paper proposes an online self-calibration method:

> Core idea: Exploit the generative-discriminative gap inside an LVLM to obtain reliable self-supervised signals, enabling online learning against hallucination.

Technical highlights:

1. Generative-Discriminative Gap

The same LVLM may hallucinate when generating answers
But it judges whether an answer is correct more accurately
This gap becomes the source of self-supervision

2. Self-Knowledge

The model knows what it can perceive clearly
It does not operate beyond its capabilities
When uncertain, it says "uncertain" instead of guessing

3. Online Learning

Not offline supervision from a stronger model
Real-time self-calibration, adjusting via the generative-discriminative gap
Continuous improvement

4. No External Supervision Needed

No dependence on a stronger model
Self-supervised, more reliable, more economical

An analogy:

Traditional methods = a teacher tells the student the answer; the teacher sees more clearly, so the student memorizes answers beyond their own understanding
New method = the student learns to check their own answers: "Am I sure about this part? This part I'm unsure about" - honestly facing one's own abilities

Why Self-Calibration Beats External Supervision

Problems with external supervision:

Capability mismatch: stronger models perceive more detail; weaker models are forced to align beyond their perception and can only guess
Unreliable: the stronger model can also be wrong; erroneous signals propagate and corrupt the student
Expensive: requires a strong model to generate supervision; high compute cost; no online updating

Advantages of self-calibration:

Self-knowledge: knows its boundaries, stays within them, is honest
Online updates: real-time calibration, continuous improvement, adapts to new data
Economical: no strong teacher model needed, self-supervised, low cost

Takeaways

If you are tackling VLM hallucination, ask yourself:

1. Does my model operate beyond its perceptual capacity? 2. Am I exploiting the generative-discriminative gap? 3. Is self-supervision more reliable than external supervision here? 4. Does the model have "self-knowledge"?

The paper's key message: fighting hallucination is not about making the model "say more," but making it "speak more honestly."

When a VLM learns self-knowledge, it transforms from a rambling storyteller into an honest observer. In the future of AI perception, the best model is not the most talkative one, but the most honest one. Knowing one's own boundaries is the most reliable compass in the ocean of knowledge.