VAMPS: Visual-Assisted Mathematical Problem Solving Benchmark for Multimodal LLMs

Forum topic · 小凯 · 2026-06-05

Summary

VAMPS (Visual-Assisted Mathematical Problem Solving) is a graph-assisted mathematics benchmark introduced by Dabiriaghdam, Vassef, and Bakhtiari (arXiv:2506.00632, June 2025). It contains 1,168 multimodal, bilingual multiple-choice question-answer pairs drawn from Iranian University Entrance Exam algebra and calculus problems, expanded with human-reviewed LLM-generated synthetic variants. Problems were selected so that plotting naturally reveals solution strategies such as intersections, extrema, and asymptotes. Unlike prior multimodal benchmarks that evaluate reasoning over fixed visual inputs, VAMPS tests whether models can benefit from constructing their own useful graphs and grounding answers in visualized results. The study addresses a key gap: multimodal LLMs increasingly handle complex reasoning, but performance often degrades when they must externalize a problem through a tool and reason over its output—a situation common in real engineering and scientific workflows that rely on visualization. Across a range of models, direct analytical solving unexpectedly outperformed tool-assisted visual solving, even on problems where plotting is the natural strategy. VAMPS is designed for both benchmarking and diagnosis of tool-use and visual reasoning in multimodal LLMs.

VAMPS: Visual-Assisted Mathematical Problem Solving Benchmark

Field: NLP / Multimodal Large Language Models
Authors: Amirhossein Dabiriaghdam, Shayan Vassef, Mohammadreza Bakhtiari
Published: 2025-06-01
arXiv: 2506.00632

Abstract

Multimodal large language models are increasingly capable of complex reasoning, yet their performance often degrades when they must externalize a problem through a tool and then reason over the tool's output—specifically when they rely on visual aids. This gap is especially important because real engineering and scientific workflows often rely on visualization tools for analysis, validation, and decision-making.

To study this discrepancy, the authors introduce VAMPS (Visual-Assisted Mathematical Problem Solving), a benchmark for graph-assisted mathematics. VAMPS contains 1,168 multimodal, bilingual multiple-choice question-answer pairs drawn from Iranian University Entrance Exam algebra and calculus problems and expanded with human-reviewed LLM-generated synthetic variants. All problems were selected so that plotting provides a natural solving strategy by revealing features such as intersection points, extrema, and asymptotes.

Key Points

1,168 multimodal, bilingual multiple-choice question-answer pairs (algebra and calculus).
Problems curated so that graphing is a natural solution strategy (intersections, extrema, asymptotes).
Unlike prior multimodal benchmarks that evaluate reasoning over fixed visual inputs, VAMPS tests whether models can construct useful graphs themselves and ground their answers in the visualization results.
Designed for both benchmarking and diagnostic analysis of tool use and visual reasoning.
Main finding: across a variety of models, direct analytical solving unexpectedly outperforms tool-enabled visual solving, even on problems where plotting is the natural strategy.

Relevance

VAMPS highlights a practical weakness in current multimodal LLMs: the ability to leverage external visualization tools during reasoning does not yet match their internal analytical capabilities, despite the importance of such workflows in real engineering and scientific practice.

*Source: arXiv:2506.00632*