Self-Evolving Visual Questioner: Teaching AI to Ask Better Questions Without Human Labels

Forum topic · 小凯 · 2026-06-18

Summary

A detailed breakdown of the paper "Self-Evolving Visual Questioner" (arXiv:2606.13929) by researchers from University of Maryland, UCLA, Peking University, Arena, and MBZUAI. The work addresses a neglected gap: vision-language models (VLMs) are trained to answer questions but rarely to ask good ones. The authors propose a fully self-supervised evolutionary framework that requires no human annotation, external teacher models, or reward models. Each round involves three steps—multi-intent question proposal, question rewriting by the initial model M0 (acting as a pre-adapted evolution operator), and quality filtering—followed by dual-format training that combines question generation (image to question+answer) and question answering formats. After two evolution rounds on Qwen2.5-VL-3B, question-generation quality improved about 82% (average score 0.25 to 0.45), with spatial reasoning rising from near zero, while QA accuracy slightly improved rather than degrading. The paper also introduces an agentic evaluation protocol covering perception difficulty, reasoning difficulty, and questioning diversity. Experiments show that 10K self-supervised samples outperform raw dataset annotations, and improved questioners produce better downstream QA supervision.

Self-Evolving Visual Questioner: Teaching AI to Ask Better Questions

Paper: Self-Evolving Visual Questioner Authors: Yijun Liang, Hengguang Zhou, Ming Li, Lichen Li, Cho-Jui Hsieh, Tianyi Zhou Institutions: University of Maryland, UCLA, Peking University, Arena, MBZUAI Links: https://arxiv.org/abs/2606.13929 | https://joliang17.github.io/SelfEvolvingVQG/

The Problem: VLMs Answer but Don't Ask

Current vision-language models (VLMs) are almost exclusively "passive answerers": humans provide questions, models provide answers. Question generation (VQG) methods remain stuck on static data distributions—relying on human annotations or stronger external models—so generated questions collapse into repetitive templates about salient objects and surface recognition.

The paper's key insight: naive self-training does not automatically yield self-improvement. Without explicit mechanisms to (1) maintain exploratory diversity, (2) increase question difficulty, and (3) ensure visual grounding, a model reinforces its own biases and collapses into repetitive, low-information questions.

The Framework: Propose → Rewrite → Filter

Each evolution round t, starting from the current model M(t):

1. Question Proposal: M(t) generates candidate questions per unlabeled image across multiple visual intents (direct recognition, comparison, spatial relations, scene understanding, grounded reasoning), each with its own answer. 2. Question Rewriting: The *initial* model M0—not M(t)—rewrites proposals to be harder and more visually centric, guided by evolution instructions (visual inspection difficulty, evidence grounding, contextual reasoning, spatial reasoning). Using M0 avoids tightly coupled proposal-rewrite distributions and injects diverse exploration. 3. Question Filtering: Candidates are kept only if visually answerable, visually grounded, and improved in perception or reasoning difficulty over the original proposal.

Dual-Format Training

Training data combines:

QG format (image → question + answer): trains the "asking muscle"
QA format (image + question → answer): anchors answering behavior

This prevents drift away from downstream VQA capability while improving question quality. 10K self-supervised samples suffice.

The filtered data fine-tunes M(t) into M(t+1), which becomes the next round's proposer—creating a closed self-evolution loop over the same unlabeled image pool.

Evaluation Protocol

Beyond BLEU/CIDEr or QA accuracy, the paper uses an agentic evaluation with an image-conditioned judge (GPT-5.4, blind to model identity), scoring 0–5 per dimension:

Perception difficulty: visual search difficulty; visual evidence coverage
Reasoning difficulty: visual context reasoning; visual spatial reasoning
Questioning diversity: semantic distance among questions on the same image (via Qwen3-Embedding-4B)

Key Results

Backbones tested: Qwen2.5-VL-3B, Qwen2.5-VL-7B, Qwen3VL-4B.

| Model | Round | QG Avg | QA Avg | |---|---|---|---| | Qwen2.5-VL-3B | Base | 0.25 | 61.90% | | | Round 2 | 0.45 (+80%) | 62.78% (+1.4%) | | Qwen3VL-4B | Base | 0.38 | 68.92% | | | Round 2 | 0.48 (+26%) | 69.50% (+0.8%) |

Findings:

All dimensions improve over two rounds, with spatial reasoning up +1133% (from ~0.03 to 0.37) on Qwen2.5-VL-3B.
Ablation on formats: QA-only scores 0.34 QG; QG-only scores 0.42 QG but *drops* QA to 61.23%; QA+QG achieves the best balance (0.45 QG, 62.78% QA).
Ablation on pipeline: self-generation alone reaches 0.35; adding rewriting reaches 0.40; adding filtering reaches 0.45—rewriting expands the candidate pool, filtering selects high-quality supervision.
Self-supervised vs raw labels: 10K self-supervised samples (0.45) beat 10K raw SAT annotations (0.36); mixing them dilutes the signal (0.42).
Better questions → better supervision: QA training with improved questions raises average accuracy from 61.90% to 63.32%, notably CVBench-3D from 69.25% to 75.58%.

Why It Works

1. Separated proposal and rewriting roles: M(t) explores new question space; M0 acts as a pre-adapted evolution operator, preventing collapse—analogous to a GAN-style split using different checkpoints of the same model. 2. Filtering enforces difficulty growth: kept candidates must exceed original proposals in difficulty, forcing steady evolution. 3. Explicit diversity objective prevents mode collapse in self-training.

Limitations and Future Directions

Filtering criteria could be finer-grained (grounding verifiers, ambiguity detectors)
Higher compute cost than training on existing QA data
Evaluated only on 2D images; video, 3D, and multi-image settings unexplored
Judge dependence on GPT-5.4 may introduce bias
Future: agent-style active questioning, curriculum-based evolution, cross-modal transfer

Takeaways

Asking and answering should be trained together—they are complementary, not zero-sum.
Self-evolution needs an "evolution operator" (the M0 rewriter) to avoid self-training collapse.
Diversity must be explicitly optimized, not left as a byproduct.
Quality of questions beats quantity: better questioners produce better downstream supervision.

> *"We show that a VLM can continuously improve itself as a visual questioner without any external supervision."* — core claim of the paper

Reference: arXiv:2606.13929 | joliang17.github.io/SelfEvolvingVQG/