OpenVLHarness: Real-World Visual Agentic Tool-Use Framework with Persistent Multimodal Memory

Forum topic · 小凯 · 2026-10-21

Summary

OpenVLHarness is a multimodal reasoning framework introduced by researchers including Yu Zhou, Kai-Wei Chang, and Nanyun Peng (arXiv 2610.26860). The paper observes that multimodal large language models (MLLMs) often cannot solve visual reasoning tasks through internal inference alone: a controlled study across 22 benchmarks shows that merely increasing the base model's inference budget yields only modest gains. Tasks instead require grounded perception, external knowledge, and numerical measurements from heterogeneous tools. OpenVLHarness addresses this with two components: a capability layer that combines task-level tool interfaces with evidence rendering for consistent access to diverse backends, and persistent multimodal memory that retains images, structured data, and web references as addressable artifacts across reasoning steps. Evaluated on 22 benchmarks with 8 open-source and proprietary backbone models, OpenVLHarness improves average scores by 9.7–12.6 points over tool-free baselines, including +11.2 for Qwen3-VL-8B, +9.7 for Kimi K3, and +9.9 for GPT-6 Astra.

Paper Overview

Research Area: NLP Authors: Yu Zhou, Cheng-Fu Yang, Rui Sun, Di Wu, Sibo Peng, Mingyang Zhang, Zhen Yang, Ruiyi Zhang, Nanyun Peng, Zhe Gan, Kai-Wei Chang Published: 2026-10-19 arXiv: 2610.26860

Summary

Multimodal large language models (MLLMs) have made rapid progress on visual reasoning, yet many tasks require evidence that is difficult to obtain through internal reasoning alone: the authors' controlled study on 22 benchmarks shows that simply increasing the inference budget of the base model yields only modest gains. Such tasks demand grounded perception, external knowledge, and numerical measurements obtained through heterogeneous tools, whose outputs must be made interpretable to the model and retained across reasoning steps.

Key Contributions

OpenVLHarness: a multimodal reasoning harness organized around two components:
Capability layer: combines task-level tool interfaces with evidence rendering, providing consistent access to heterogeneous backends.
Persistent multimodal memory: retains images, structured data, and web references as addressable artifacts across reasoning steps.

Results

Evaluated on 22 benchmarks using 8 open-source and proprietary backbone models, OpenVLHarness improves average scores by 9.7–12.6 points over tool-free baselines:

Qwen3-VL-8B: +11.2
Kimi K3: +9.7
GPT-6 Astra: +9.9

OpenVLHarness: Real-World Visual Agentic Tool-Use Framework with Persistent Multimodal Memory

Summary

Paper Overview

Summary

Key Contributions

Results

Links

Tags