Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Visual Agents
- Field: Computer Vision (CV)
- Authors: Haozhi Yuan, Handong Li, Yifan Zhang, Qianshan Wei
- Published: 2026-11-09
- arXiv: 2611.12886
- Stage 1 — GD-GRPO: Trains spatial and temporal experts using evidence-aware rewards combined with stochastic removal of raw visual inputs, pushing the model to acquire grounded evidence and actually leverage tool-returned observations.
- Stage 2 — OG-MOPD: Replaces defective tool actions with canonical ones midway through rollouts, then resumes distillation from the repaired observations.
- Synergy: The two stages are complementary — learning to rely on tool-returned observations makes the policy better positioned to benefit from observation repair.
- No cold start: Both stages eliminate the need for task-specific SFT cold starts.
- Extensive experiments on 14 image and video benchmarks show Reframe outperforms both Mix-RL and the original MOPD.
- Ablation studies indicate GD-GRPO significantly improves observation reliance and tool gains compared with outcome-based GRPO.
- Paper: https://arxiv.org/abs/2611.12886
Overview
Agentic reinforcement learning (RL) and multi-teacher on-policy distillation (MOPD) offer complementary routes to post-training. For visual agents, however, tool use presents two related challenges. Models may overlook useful tool-returned observations, while poorly grounded actions can limit the evidence available for subsequent distillation.
Reframe is a post-training framework that treats the observation state, rather than only the output distribution, as an object of alignment.