Paper Overview
- Field: Computer Vision (CV)
- Authors: Junhao Cheng, Liang Hou, Tianxiong Zhong
- arXiv: 2506.00010
- Textual descriptions cannot capture intricate spatiotemporal details.
- VGMs struggle to faithfully execute fine-grained or long-tail instructions even with a valid plan.
- +16.7 average performance improvement over baselines
- Substantially outperforms the VLM-as-Solver paradigm (+0.4) and Best-of-N scaling (+2.2) at comparable test-time cost
- Project page: https://VLM-as-Teacher.github.io/
- Paper: https://arxiv.org/abs/2506.00010
Background
The recent "Reasoning with Video" paradigm uses Video Generation Models (VGMs) to generate temporally coherent visual trajectories for reasoning tasks. Despite strong visual quality, state-of-the-art VGMs often fail to understand and follow task-specific rules, causing logical failures across diverse reasoning scenarios.
Prior work uses Vision-Language Models (VLMs) as problem pre-solvers that produce or refine textual guidance for the VGM. However, this approach has two limitations:
Key Idea: VLM as Teacher
While VLMs struggle as solvers, they possess strong perception capabilities to evaluate process-constraint satisfaction and final-goal achievement. The paper leverages this strength by shifting the VLM's role to that of a teacher:
1. The VLM teacher extracts task-specific rules and constructs differentiable rewards. 2. These rewards guide test-time online optimization of the VGM reasoner via lightweight LoRA modules. 3. This enables adaptive test-time optimization, extending reasoning capabilities beyond the VGM's inherent boundaries.
Results
Evaluated on symbolic (VBVR-Bench) and general (RULER-Bench) video reasoning benchmarks:
Conclusion
Integrating VLMs as test-time teachers offers a promising paradigm for generalizable video reasoning.