Self-Evolving Visual Questioner: Teaching AI to Ask Better Questions Without Any Supervision

Paper: Self-Evolving Visual Questioner Authors: Yijun Liang, Hengguang Zhou, Ming Li, Lichen Li, Cho-Jui Hsieh, Tianyi Zhou Affiliations: University of Maryland, UCLA, Peking University, Arena, MBZUAI arXiv: https://arxiv.org/abs/2606.13929 Project page: https://joliang17.github.io/SelfEvolvingVQG/

The Problem

Most VLMs are passive answerers: humans provide questions, models provide answers. Their questioning ability—actively inspecting an image, identifying valuable visual evidence, and constructing questions requiring meaningful perception and reasoning—is underdeveloped. Existing visual question generation (VQG) methods are stuck mimicking static templates from human annotations or stronger external models, producing repetitive, surface-level questions about salient objects.

Key Insight: Naive Self-Training Collapses

Self-training does not automatically produce self-improvement. Without explicit mechanisms to (1) preserve exploration diversity, (2) increase question difficulty, and (3) ensure visual grounding, a model reinforces its own biases and collapses into repetitive, low-information, weakly grounded questions.

The Three-Step Evolution Loop

At each round t, the current model Mt:

1. Question Proposal: For each unlabeled image, Mt generates candidate questions under different visual intents (direct recognition, comparison, spatial relations, scene understanding, grounded reasoning), answering each itself. 2. Question Rewriting: The initial model M0 (not Mt) rewrites questions toward harder directions: more visual inspection, stronger evidence grounding, contextual or spatial reasoning. Using M0 as a "pre-adapted evolution operator" decouples proposal and rewriting distributions and prevents collapse. 3. Question Filtering: Keep only candidates that are visually answerable, visually grounded, and harder than the original proposal; discard ambiguous, weakly grounded, or unimproved ones.

Filtered data is then fine-tuned in dual formats:

• QG format (image → question + answer): trains the questioning ability.
• QA format (image + question → answer): anchors answering behavior, preventing drift.

The resulting model Mt+1 becomes the next round's proposer, closing the self-evolution loop.

Agentic Evaluation Protocol

Beyond BLEU/CIDEr or QA accuracy, the paper evaluates questions on:

• Perception difficulty: visual search difficulty, visual evidence coverage
• Reasoning difficulty: visual context reasoning, visual spatial reasoning
• Diversity: semantic non-redundancy across questions on the same image (via Qwen3-Embedding-4B)

GPT-5.4 serves as a blind, image-conditioned judge scoring each dimension 0–5.

Results

Tested on Qwen2.5-VL-3B/7B and Qwen3VL-4B using ~10K self-supervised samples:

• Qwen2.5-VL-3B: QG average rose 0.25 → 0.45 after two rounds (~80–82% gain). Spatial reasoning improved from 0.03 to 0.37 (+1133%). QA accuracy rose from 61.90% to 62.78%.
• Qwen3VL-4B: QG average 0.38 → 0.48 (+26%), QA 68.92% → 69.50%.
• Ablations: QA+QG dual-format beats QA-only or QG-only (QG-only degrades answering to 61.23%). Rewriting and filtering are complementary (0.35 → 0.40 → 0.45). Self-generated data beat a 10K sample of original SAT annotations (0.45 vs 0.36 QG average), and mixing diluted the signal.
• Better questions → better supervision: training on questions from the evolved model raised downstream QA accuracy from 61.90% to 63.32%, with CVBench-3D jumping from 69.25% to 75.58%.

Why It Works

1. Proposal/rewriting separation — Mt explores new question spaces while M0 provides an independent perspective (analogous to GAN generator-discriminator separation, but across checkpoints of the same model). 2. Filtering enforces progress — candidates must exceed their proposals in difficulty, steering evolution upward. 3. Dual-format synergy — better questioning improves visual inspection and answering; better answering improves filtering.

Limitations and Future Work

• Filtering could be finer-grained (grounding verifiers, ambiguity detectors)
• The propose–rewrite–filter–train loop is computationally heavier than training on existing QA data
• Only 2D images evaluated; video, 3D, and multi-image settings remain unexplored
• GPT-5.4 as judge may introduce bias

Future directions include agentic VLMs that actively question their environment, curriculum-based difficulty adjustment, and cross-modal transfer.

Takeaways

1. Questioning and answering can be jointly trained and win together — not a zero-sum tradeoff. 2. Self-evolution needs an evolution operator — M0's role prevents tightly-coupled self-training collapse. 3. Quality beats quantity — 10K high-quality self-supervised samples outperformed real annotations at the same budget. 4. Closed-loop self-improvement is feasible — a VLM can continuously improve a specific capability with zero external supervision.

> "We show that a VLM can continuously improve itself as a visual questioner without any external supervision."

Reference: arXiv:2606.13929 | https://joliang17.github.io/SelfEvolvingVQG/