English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original
智柴网 中文原文

Detectives, Math, and Code: How AIR Enables Sherlock-Style Interleaved Reasoning in Multimodal AI

Forum topic · 小凯 · 2026-06-23

Summary

AIR (Adaptive Interleaved Reasoning with Code in MLLMs) is a training framework that teaches multimodal large language models to alternate between visual perception, note-taking, external queries, and precise code-based calculation—much like Sherlock Holmes working a crime scene. The method has three pillars: (1) a two-stage cold-start pipeline that generates seed chain-of-thought data from a teacher model, then refines it via code execution verification, step decomposition, and diversity augmentation; (2) RL dataset curation with correctness, difficulty, and diversity filtering to keep only problems solvable with tools; and (3) a group-constrained reward function that scores reasoning process quality and compares multiple tool-use strategies per question, letting models learn when to invoke tools adaptively rather than via hand-crafted heuristics. Experiments on MathVista, MathVision, and MMMU report a 6.1-point average gain after RL training, a 9.9-point improvement on interleaved-reasoning-heavy samples, and over 95% tool-call success. The post also discusses limitations, including dependence on teacher models, a limited tool ecosystem, and code execution safety concerns.

Detectives, Math, and Code: How AIR Enables Sherlock-Style Interleaved Reasoning in Multimodal AI

> *"When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth."* — Sherlock Holmes

This post is an English translation/editorial summary of a Chinese forum article explaining the paper AIR: Adaptive Interleaved Reasoning with Code in MLLMs (arXiv:2606.23678).

The Big Idea

At the end of 2024, OpenAI's o3 showcased a new way of thinking: interleaved reasoning—not answering in one shot, but switching repeatedly between perception, recording, querying, and computation. Imagine Holmes at a crime scene: he observes (visual perception), notes three cigar butts of two brands (recording), measures footprints on the windowsill (measurement), checks rainfall records (external query), and finally draws a timeline to deduce when the killer left (calculation and reasoning).

Current multimodal large language models (MLLMs) like GPT-4V, Claude 3, and Gemini can describe photos, recognize objects and text, and interpret charts—but they fail at mathematical computation. Training data separates vision from calculation: models have seen countless charts and countless math problems, but rarely "doing math while looking at a chart."

Limitations of Prior Work

Existing attempts to add tool use to MLLMs share two flaws:

1. Predefined heuristic rules — e.g., "if it's a math problem, call the calculator." Real cases don't follow a manual. 2. Focus on visual operations, ignoring numerical computation — models can "turn the cat in the image into a dog" but can't compute the average of a bar chart.

AIR's Three-Component Solution

1. Cold-Start Data Construction

A two-stage pipeline:

  • Stage 1 — Seed data generation: a capable teacher model (e.g., GPT-4V) generates chain-of-thought solutions to many visual-math problems.
  • Stage 2 — Refinement and expansion:
  • Execution verification: actually run the generated code; flag data where execution results mismatch claimed results.
  • Step decomposition: break long reasoning chains into fine-grained steps (observe, compute, query, verify).
  • Diversity augmentation: generate multiple solution paths for the same problem.

    • 2. RL Dataset Curation

    A multi-layer filtering system:

  • Correctness filtering: keep only data where code execution matches the final answer.
  • Difficulty filtering: use a difficulty scorer; discard problems too hard or too easy, keeping the "sweet spot" where tools are required.
  • Diversity filtering: ensure coverage of different reasoning patterns and encourage exploratory paths.

    • 3. Adaptive Tool-Calling via Group-Constrained Reward

    Instead of a simple right/wrong reward, AIR uses:

  • Process rewards: credit for correct visual extraction, appropriate tool calls, successful code execution, coherent intermediate steps, and the correct final answer.
  • Group constraints: for each question, generate multiple reasoning trajectories (e.g., 8) grouped by different tool-use strategies; if a group performs better overall, all trajectories in that group receive boosted rewards. This teaches strategy selection, not just individual skills.

    • The result is an emergent "intuition": generate code for complex numerical problems, run queries to verify hypotheses, request more visual detail when information is insufficient.

    Experimental Results

    Evaluated on MathVista, MathVision, and MMMU:

  • +6.1 percentage points average performance after RL training.
  • +9.9 percentage points on interleaved-reasoning samples requiring multi-step reasoning and tool calls.
  • Over 95% of tool calls executed successfully with valid results.

    • Ablations show:

  • Without cold-start data, RL from scratch is unstable and models often collapse.
  • Group-constrained rewards significantly outperform result-only rewards, especially on complex reasoning.
  • Unfiltered data slows training and injects noise.

    • Significance

  • From pattern recognition toward genuine reasoning: decomposing problems, invoking tools, verifying intermediate results.
  • A milestone for general tool use: RL-trained tool strategies rather than hand-coded rules mean new tools can be adapted quickly.
  • A step toward AGI-style cognition: humans also offload computation to paper, calculators, and references.

    • Potential Applications

  • Scientific discovery: extract data from experiment charts, run statistics, generate visualizations.
  • Financial analysis: extract metrics from statement screenshots, compute ratios and growth rates.
  • Education: solve homework photos step by step with tool-assisted verification.

    • Limitations and Challenges

  • Dependence on the teacher model for cold-start data; teacher biases propagate.
  • Limited tool ecosystem: mainly Python execution; real-world use needs databases, APIs, MATLAB/R, etc.
  • Safety risks: auto-generated executed code requires strict sandboxing and permission controls.

    • Future Directions

  • Interleaving more modalities: audio+code, video+code, sensor data+code.
  • Metacognition: letting AI audit its own reasoning ("my assumption may be wrong, let me re-check").
  • Collaborative reasoning across multiple agents (visual analysis, computation, logical verification).

    • Conclusion

    AIR's lesson: intelligence is not just knowing a lot—it's knowing how to effectively use what you know. Holmes's greatness lay not in memorizing every poison formula, but in knowing when to use the microscope, when to consult references, and when to reason logically. AIR teaches AI the same skill.

    References

  • Han, C., Lan, X., & Qiu, H. (2026). AIR: Adaptive Interleaved Reasoning with Code in MLLMs. arXiv:2606.23678.
  • OpenAI. (2024). Introducing OpenAI o3 and o3-mini. https://openai.com/index/openai-o3-system-card/
  • Lu, P., et al. (2023). MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. arXiv:2310.02255.
  • Yue, X., et al. (2023). MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark. arXiv:2311.16502.
  • Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022.
  • Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022.

Tags

#multimodal-ai#reinforcement-learning#reasoning#tool-use#mllm#code-execution#mathvita#arxiv-paper

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178208052