Vision-OPD: Self-Distillation to Improve Fine-Grained Visual Understanding in Multimodal LLMs

Forum topic · 小凯 · 2026-05-20

Summary

Vision-OPD (Vision On-Policy Distillation) is a regional-to-global self-distillation framework that improves fine-grained visual understanding in Multimodal Large Language Models (MLLMs). The authors observe a regional-to-global perception gap: the same MLLM answers fine-grained questions more accurately when conditioned on evidence-centered crops than on corresponding full images, indicating that failures stem from difficulty focusing on relevant evidence rather than weak local recognition. To address this, Vision-OPD instantiates two conditional policies from the same MLLM instance: a crop-conditioned teacher and a full-image-conditioned student. The student generates on-policy rollouts, and the method minimizes token-level divergence between the teacher's and student's next-token distributions on these rollouts, transferring privileged regional perception to the full-image policy. The approach requires no external teacher model, ground-truth labels, reward verifiers, or inference-time tool use. Experiments across multiple fine-grained visual understanding benchmarks show Vision-OPD models achieve performance comparable to or better than larger open-source, closed-source, and 'Thinking-with-Images' agentic models. Paper: arXiv 2505.14302, by Qianhao Yuan, Jie Lou, and Xing Yu.

Paper Overview

Field: NLP Authors: Qianhao Yuan, Jie Lou, Xing Yu Published: 2026-05-19 arXiv: 2505.14302

Abstract

Multimodal Large Language Models (MLLMs) still struggle with fine-grained visual understanding, where answers often depend on small but decisive evidence in the full image. The authors observe a regional-to-global perception gap: the same MLLM answers fine-grained questions more accurately when conditioned on evidence-centered crops than on the corresponding full images, suggesting that many failures stem from difficulty focusing on relevant evidence rather than insufficient local recognition ability.

Motivated by this observation, the paper proposes Vision-OPD (Vision On-Policy Distillation), a regional-to-global self-distillation framework that transfers the model's own privileged regional perception to its full-image policy.

Method

Vision-OPD instantiates two conditional policies from the same MLLM: a crop-conditioned teacher and a full-image-conditioned student.
The student generates on-policy rollouts, and Vision-OPD minimizes the token-level divergence between the teacher's and the student's next-token distributions on these rollouts.
This enables the model to internalize the benefits of visual zooming without:
External teacher models
Ground-truth labels
Reward verifiers
Inference-time tool use

Results

Experiments on multiple fine-grained visual understanding benchmarks show that Vision-OPD models achieve performance comparable to or better than larger open-source, closed-source, and 'Thinking-with-Images' agentic models.

---

*Auto-collected on 2026-05-20*