VLMs as Teachers: Adaptive Test-Time Optimization for Video Reasoning

Forum topic · 小凯 · 2026-06-03

Summary

This paper introduces a paradigm shift in video reasoning by repositioning Vision-Language Models (VLMs) from solvers to teachers. Recent 'Reasoning with Video' approaches use Video Generation Models (VGMs) to produce temporally coherent visual trajectories, but VGMs often fail to follow task-specific rules. Prior work uses VLMs as pre-solvers generating textual guidance, yet text cannot capture intricate spatiotemporal details and VGMs struggle to execute fine-grained instructions. Instead, the proposed method exploits VLMs' strong perception to evaluate process-constraint satisfaction and goal achievement: the VLM teacher extracts task-specific rules to construct differentiable rewards that guide the VGM reasoner via test-time online optimization with lightweight LoRA modules. On symbolic (VBVR-Bench) and general (RULER-Bench) video reasoning benchmarks, the approach achieves an average +16.7 point improvement, substantially outperforming VLM-as-Solver (+0.4) and Best-of-N scaling (+2.2) at comparable test-time cost. Paper: arXiv:2506.00010.

Overview

Field: Computer Vision (CV)
Authors: Junhao Cheng, Liang Hou, Tianxiong Zhong
arXiv: 2506.00010
Project page: https://VLM-as-Teacher.github.io/

Key Points

Recent 'Reasoning with Video' paradigms use Video Generation Models (VGMs) to generate temporally coherent visual trajectories for reasoning tasks. Despite strong visual quality, VGMs struggle to understand and follow task-specific rules, causing logical failures.
Existing methods use Vision-Language Models (VLMs) as problem pre-solvers to generate or refine textual guidance. However, text descriptions cannot capture intricate spatiotemporal details, and VGMs often fail to faithfully execute fine-grained or long-tail instructions even with a valid plan.
The paper's insight: while VLMs struggle as solvers, they possess strong perception capabilities to evaluate process-constraint satisfaction and final-goal achievement.
Paradigm shift: VLMs act as 'teachers' — they extract task-specific rules to construct differentiable rewards that guide the VGM reasoner through test-time online optimization using lightweight LoRA modules.
This enables adaptive test-time optimization and extends reasoning capability beyond the VGM's inherent boundaries.
Results: On symbolic (VBVR-Bench) and general (RULER-Bench) video reasoning benchmarks, the method achieves an average improvement of 16.7 points, substantially outperforming VLM-as-Solver (+0.4 points) and Best-of-N scaling (+2.2 points) at comparable test-time cost.

Abstract (translated)

The recent 'Reasoning with Video' paradigm utilizes Video Generation Models (VGMs) to generate temporally coherent visual trajectories to complete reasoning tasks. Although state-of-the-art VGMs excel at visual quality, they often struggle to understand and follow task-specific rules, leading to logical failures across diverse reasoning scenarios. Existing efforts try to utilize Vision-Language Models (VLMs) as problem pre-solvers to produce or refine textual guidance for the VGM. However, textual descriptions fail to capture intricate spatiotemporal details, and VGMs often struggle to faithfully execute fine-grained or long-tail instructions even with a valid plan. Leveraging VLMs' perception capabilities, this work introduces a paradigm shift that positions VLMs as teachers, guiding VGM reasoning via differentiable rewards and adaptive test-time optimization.

*Auto-collected on 2026-06-03.*