Online Self-Calibration Against Hallucination in Vision-Language Models

> Paper: Online Self-Calibration Against Hallucination in Vision-Language Models > Authors: Minghui Chen, Chenxu Yang, Hengjie Zhu, Dayan Wu, Zheng Lin, Qingyi Si > arXiv: 2605.00323 | 2026-04-29

1. The Hallucination Problem

Imagine using an AI vision-language model on an image:

User uploads an image:

• The image contains a cat.

AI description:

• "The image shows a cat sitting on a red sofa, next to a cup of coffee, with rain visible through the window..."

Actual image:

• Only a cat. No sofa. No coffee. No window.

This is hallucination.

Existing solutions:

• Use a stronger model (e.g., GPT) to generate supervision signals
• Offline training

Problems with this approach:

• The student model is forced to align with details beyond its perceptual capability
• It fails to learn to "see"
• It learns to "guess"
• Supervision-perception mismatch

2. Online Self-Calibration: Knowing What the Model Actually "Sees"

The paper proposes an online self-calibration method.

Core idea:

> Exploit the generative-discriminative gap inside the LVLM itself to obtain reliable self-supervised signals, enabling online learning against hallucination.

Technical approach:

1. Generative-Discriminative Gap

• The same LVLM may hallucinate when *generating* an answer
• But is more accurate when *discriminating* whether an answer is correct
• This gap is the source of the self-supervised signal

2. Self-Knowledge

• The model knows what it "can clearly see"
• It does not exceed its own capability
• When uncertain, it says "uncertain" instead of guessing

3. Online Learning

• Not offline supervision from a stronger model
• Real-time self-calibration
• Adjusts based on the generative-discriminative gap
• Continuously improves

4. No External Supervision Required

• Does not depend on a stronger model
• Self-supervised
• More reliable and more economical

An analogy:

• Traditional method = a teacher tells the student "the answer is this"
• The teacher sees more clearly, so the student is forced to memorize answers beyond their own understanding
• New method = the student learns to check their own answers
• "Am I sure about this part?"
• "This part, I'm not sure about"
• Honestly facing one's own capabilities

3. Why Self-Calibration Beats External Supervision

Problems with external supervision:

• Capability mismatch: the strong model perceives more detail; the weak model is forced to align beyond its perceptual ability and can only guess
• Unreliability: the strong model can also be wrong; erroneous signals propagate and corrupt the student
• Cost: requires a strong model to generate supervision, high compute cost, no online updates

Advantages of self-calibration:

• Self-knowledge: knows its own boundaries, stays within them, is honest
• Online updates: real-time calibration, continuous improvement, adapts to new data
• Economy: no strong model needed, self-supervised, low cost

4. Takeaway

As Feynman put it:

> "Knowing what not to do is as important as knowing what to do."

In AI perception:

> Getting a model to say "I'm not sure" is more honest and more reliable than having it guess an answer. The insight of online self-calibration is that a model should not be forced to "pretend to see" things that do not exist—it should learn self-knowledge: knowing its perceptual boundaries and expressing itself honestly within them.

If you are working on VLM hallucination, ask yourself:

1. "Does my model exceed its own perceptual capability?" 2. "Am I exploiting the generative-discriminative gap?" 3. "Is self-supervision more reliable than external supervision here?" 4. "Does my model have self-knowledge?"

This paper reminds us: fighting hallucination is not about making the model "say more" but making the model "speak more honestly."

When a VLM learns self-knowledge, it transforms from a "rambling storyteller" into an "honest observer." In the future of AI perception, the best model is not the most talkative—it is the most honest.

> Note: The arXiv identifier (2605.00323) is quoted as given in the original post.