[论文] OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memo...

论文概要

研究领域: NLP 作者: Yu Zhou, Cheng-Fu Yang, Rui Sun, Di Wu, Sibo Peng, Mingyang Zhang, Zhen Yang, Ruiyi Zhang, Nanyun Peng, Zhe Gan, Kai-Wei Chang 发布时间: 2026-10-19 arXiv: 2610.26860

中文摘要

多模态大语言模型（MLLM）在视觉推理方面进展迅速，但许多任务所需证据难以仅通过内部推理获得：我们在22个基准测试上的对照研究表明，单纯增加基础模型的推理预算只能带来有限的提升。这类任务需要基于实际感知的grounding、外部知识，以及通过异构工具获得的数值测量。我们提出OpenVLHarness——一个围绕能力层和持久化多模态记忆构建的多模态推理框架。能力层将任务级工具接口与证据渲染相结合，为异构后端提供一致的访问方式；持久化多模态记忆将图像、结构化数据和网页引用保留为可寻址的工件。在22个基准测试上使用8个开源和专有骨干模型评估，OpenVLHarness的平均得分比无工具基线提高9.7-12.6分，其中Qwen3-VL-8B提升+11.2，Kimi K3提升+9.7，GPT-6 Astra提升+9.9。

原文摘要

Multimodal large language models (MLLMs) have made rapid progress on visual reasoning, yet many tasks require evidence that is difficult to obtain through internal reasoning alone: our controlled study on 22 benchmarks shows that simply increasing the inference budget of the base model yields only modest gains. Such tasks demand grounded perception, external knowledge, and numerical measurements obtained through heterogeneous tools, whose outputs must be made interpretable to the model and retained across reasoning steps. We introduce OpenVLHarness, a multimodal reasoning harness organized around a capability layer and persistent multimodal memory. The capability layer combines task-level tool interfaces with evidence rendering, providing consistent access to heterogeneous backends and con...

#论文 #arXiv #NLP #小凯