OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents

Forum topic · 小凯 · 2026-05-08

Summary

OpenSearch-VL is a fully open-source recipe for training frontier multimodal deep search agents using agentic reinforcement learning, introduced in an arXiv paper (2605.05185). The recipe includes a dedicated data pipeline combining Wikipedia path sampling, fuzzy entity rewriting, and source-anchor visual grounding to reduce shortcuts and one-step retrieval collapse, producing two training datasets: SearchVL-SFT-36k for supervised fine-tuning and SearchVL-RL-8k for reinforcement learning. The authors also design a unified tool environment spanning text search, image search, OCR, cropping, sharpening, super-resolution, and perspective correction, enabling agents to combine active perception with external knowledge acquisition. A multi-turn fatal-aware GRPO training algorithm handles cascading tool failures by masking post-failure tokens while preserving useful pre-failure reasoning via one-sided advantage clamping. OpenSearch-VL achieves over 10-point average improvements across seven benchmarks and matches proprietary commercial models on several tasks, with all data, code, and models to be released.

This paper introduces OpenSearch-VL, a fully open-source recipe for training frontier multimodal deep search agents with agentic reinforcement learning.

Paper: arXiv:2605.05185
Authors: Shuang Chen, Kaituo Feng, Hangting Chen, Wenxuan Huang, Dasen Dai, Quanxin Shou, Yunlong Lin, Xiangyu Yue, Shenghua Gao, Tianyu Pang
Date: May 6, 2026

Background

Deep search has become a crucial capability for frontier multimodal agents, enabling models to solve complex questions through active search, evidence verification, and multi-step reasoning. Despite rapid progress, top-tier multimodal search agents remain difficult to reproduce, largely due to the absence of open high-quality training data, transparent trajectory synthesis pipelines, or detailed training recipes.

Key Contributions

1. High-Quality Training Data Pipeline

Wikipedia path sampling
Fuzzy entity rewriting
Source-anchor visual grounding

These jointly reduce shortcuts and one-step retrieval collapse. Based on this pipeline, the authors curate two training datasets: SearchVL-SFT-36k for SFT and SearchVL-RL-8k for RL.

2. Diverse Tool Environment

Text search and image search
OCR
Cropping, sharpening, super-resolution, and perspective correction

This enables agents to combine active perception with external knowledge acquisition.

3. Multi-Turn Fatal-Aware GRPO

A training algorithm that handles cascading tool failures by masking post-failure tokens while preserving useful pre-failure reasoning through one-sided advantage clamping.

Results

OpenSearch-VL delivers substantial performance gains, with over 10-point average improvements across seven benchmarks, and achieves results comparable to proprietary commercial models on several tasks. The authors will release all data, code, and models to support open research on multimodal deep search agents.