MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents

Forum topic · 小凯 · 2026-07-15

Summary

MM-ToolSandBox is a benchmark and evaluation framework for visually grounded tool-calling agents. It provides a stateful execution environment spanning 500+ tools across 16 application domains, supporting multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls while handling realistic conversational phenomena such as goal revisions, error corrections, and state mutations. An automated scenario generation pipeline using information-flow-guided planning and multi-stage quality filtering yields 258 human-verified nominal scenarios plus 50 variants targeting interactive UI applications. Evaluation of 12 state-of-the-art models, from 4B open-weight to frontier proprietary systems, shows that current models still lack robust visual tool-calling ability: even the best model achieves below 50% success. Failure analysis reveals that visual precision, not just planning, is the main bottleneck—53% of failures stem from incorrectly extracting information from images despite otherwise correct task workflows. A planning-precision crossover emerges: smaller models fail at deciding what to do, while larger models fail at perceiving what they see, suggesting fundamentally different improvement paths across capability levels. Paper: arXiv 2607.11818.

Paper Overview

Research Area: Computer Vision (CV) Authors: Kaixin Ma, Di Feng, Alexander Metz, Jiarui Lu, Eshan Verma Published: 2026-07-13 arXiv: 2607.11818

Introduction

This paper introduces MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents. The framework provides a stateful execution environment spanning 500+ tools across 16 application domains, supporting multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls while handling realistic conversational phenomena (goal revisions, error corrections, state mutations).

Benchmark Construction

An automated scenario generation pipeline produces diverse, visually grounded scenarios through:

Information-flow-guided planning
Multi-stage quality filtering

This yields 258 human-verified nominal scenarios plus 50 variants targeting interactive UI applications.

Key Findings

Evaluation of 12 state-of-the-art models (from 4B open-weight to frontier proprietary systems) shows that current models still lack robust visual tool-calling capabilities:

Even the best model achieves below 50% success rate
Failure analysis reveals that visual precision—not just planning—is the main bottleneck: 53% of failures stem from incorrectly extracting information from images, despite otherwise correct task workflows
A planning-precision crossover emerges with scale: smaller models fail at deciding *what to do*, while larger models fail at perceiving *what they see*, implying fundamentally different improvement directions for models at different capability levels

---

*Auto-collected on 2026-07-15*