MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents

Forum topic · 小凯 · 2026-07-15

Summary

MM-ToolSandBox is a benchmark and evaluation framework for visually grounded tool-calling agents. It provides a stateful execution environment spanning 500+ tools across 16 application domains and supports multi-image, multi-turn tasks in which agents must ground progressively arriving visual inputs into executable tool calls while handling realistic conversational phenomena such as goal revisions, error corrections, and state mutations. An automated scenario generation pipeline uses information-flow-guided planning and multi-stage quality filtering, producing 258 human-verified nominal scenarios plus 50 variants targeting interactive UI applications. Evaluation of 12 state-of-the-art models, from 4B open-weight systems to frontier proprietary models, shows that even the best model achieves below 50% success rate. Failure analysis reveals that visual precision—not just planning—is a key bottleneck: 53% of failures stem from incorrect information extraction from images despite otherwise correct task workflows. A 'planning-precision crossover' emerges between scale and capability: smaller models fail at deciding what to do, while larger models fail at perceiving what they see, suggesting fundamentally different improvement paths at different capability levels.

Paper Overview

Research area: Computer Vision (CV) Authors: Kaixin Ma, Di Feng, Alexander Metz, Jiarui Lu, Eshan Verma Published: 2026-07-13 arXiv: 2607.11818

Introduction

This paper introduces MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents.

Key Features

Stateful execution environment spanning 500+ tools across 16 application domains
Multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls
Support for realistic conversational phenomena, including goal revisions, error corrections, and state mutations

Scenario Generation

An automated scenario generation pipeline produces diverse, visually grounded scenarios through:

Information-flow-guided planning
Multi-stage quality filtering

This yields 258 human-verified nominal scenarios and 50 variants targeting interactive UI applications.

Evaluation Results

The authors evaluate 12 state-of-the-art models, ranging from 4B open-weight models to frontier proprietary systems. Key findings:

Current models still lack robust visually grounded tool-calling ability — even the best model achieves a success rate below 50%.
Failure analysis shows that visual precision, not just planning, is a major bottleneck: 53% of failures originate from incorrectly extracting information from images, even when the task workflow is otherwise correct.
A "planning-precision crossover" appears between scale and capability: smaller models fail at deciding *what to do*, whereas larger models fail at perceiving *what they see*. This suggests that improvements for models at different capability levels are fundamentally different.