VAMPS: A Visual-Assisted Mathematical Problem Solving Benchmark for Multimodal LLMs

Forum topic · 小凯 · 2026-06-05

Summary

VAMPS (Visual-Assisted Mathematical Problem Solving) is a benchmark introduced by researchers to evaluate whether multimodal large language models can benefit from constructing and reasoning over their own graphs. The benchmark contains 1,168 multimodal, bilingual multiple-choice question-answer pairs based on Iranian University Entrance Exam algebra and calculus problems, expanded with human-reviewed LLM-generated synthetic variants. All problems were selected so that plotting reveals natural solution strategies such as intersections, extrema, and asymptotes. Unlike prior multimodal benchmarks that primarily evaluate reasoning over fixed visual inputs, VAMPS tests whether models can externalize problems through tools and ground their answers in the resulting visualizations. Across a range of models, the study found that direct analytical problem solving surprisingly outperforms tool-enabled visual solving, even on problems where plotting is the natural strategy. This highlights a significant gap in how multimodal LLMs handle tool use and visual reasoning, which matters for real engineering and scientific workflows that rely on visualization for analysis, validation, and decision-making. The paper is available on arXiv as 2506.00632.

Overview

Field: NLP
Authors: Amirhossein Dabiriaghdam, Shayan Vassef, Mohammadreza Bakhtiari
Published: 2025-06-01
arXiv: 2506.00632

Abstract

Multimodal large language models are increasingly capable of complex reasoning, yet their performance often degrades when they must externalize a problem through a tool and then reason over the tool's output, specifically when they rely on visual aids. This gap is especially important because real engineering and scientific workflows often rely on visualization tools for analysis, validation, and decision-making.

Key points

To study this discrepancy, the authors introduce VAMPS (Visual-Assisted Mathematical Problem Solving), a benchmark for graph-assisted mathematics.
VAMPS contains 1,168 multimodal, bilingual multiple-choice question-answer pairs drawn from Iranian University Entrance Exam algebra and calculus problems, expanded with human-reviewed LLM-generated synthetic variants.
All problems were selected so that plotting provides a natural solution strategy by revealing intersections, extrema, asymptotes, and other visual cues.
VAMPS is designed for both benchmarking and diagnosis: it goes beyond prior multimodal benchmarks that mainly evaluate reasoning over fixed visual inputs, instead testing whether models can benefit from constructing useful graphs and grounding their answers in visualization results.
Overall, the authors find that across various models, direct analytical solving surprisingly outperforms tool-enabled visual solving, even on problems where plotting is a natural strategy.

Why it matters

This result exposes a gap between models' multimodal reasoning abilities and their capacity to use visualization tools effectively — a capability central to real-world engineering and scientific workflows that depend on visualization for analysis, validation, and decision-making.