Act Wisely: Cultivating Meta-Cognitive Tool Use in Agentic Multimodal Models (HDPO/Metis)

Forum topic · 小凯 · 2026-04-12

Summary

This paper introduces HDPO, a reinforcement learning framework that addresses the meta-cognitive deficit in agentic multimodal models: the tendency to blindly invoke external tools even when queries can be resolved from raw visual context. Such reflexive tool use causes latency bottlenecks and injects noise that harms reasoning. Prior RL approaches penalize tool usage via a scalarized reward, but this coupling creates an optimization dilemma: aggressive penalties suppress necessary tool use, while mild penalties are drowned out by accuracy-reward variance during advantage normalization. HDPO reframes tool efficiency from a competing scalar objective into a strict conditional objective, maintaining two orthogonal optimization channels: an accuracy channel maximizing task correctness, and an efficiency channel enforcing execution economy via conditional advantage estimation only within accurate trajectories. This decoupled architecture induces a cognitive curriculum—agents first master task-solving before refining self-reliance. The resulting model, Metis, reduces tool calls by orders of magnitude while improving reasoning accuracy. Paper: arXiv 2504.07927 (CV, April 2025).

Paper Overview

Field: Computer Vision (CV) Authors: Shilin Yan, Jintao Tong, Hongwei Xue Published: 2025-04-10 arXiv: 2504.07927

Abstract (Translation)

The advent of agentic multimodal models has empowered systems to actively interact with external environments. However, current agents suffer from a profound meta-cognitive deficit: they struggle to arbitrate between leveraging internal knowledge and querying external utilities. Consequently, they frequently fall prey to blind tool invocation, resorting to reflexive tool execution even when queries are resolvable from the raw visual context. This pathological behavior precipitates severe latency bottlenecks and injects extraneous noise that derails sound reasoning.

Existing reinforcement learning protocols attempt to mitigate this via a scalarized reward that penalizes tool usage. Yet, this coupled formulation creates an irreconcilable optimization dilemma: an aggressive penalty suppresses necessary tool usage, while a mild penalty is entirely drowned out by the variance of the accuracy reward during advantage normalization, rendering it ineffective against tool overuse.

Key Contributions

HDPO framework: Reframes tool efficiency from a competing scalar objective into a strict conditional objective, avoiding reward scalarization.
Two orthogonal optimization channels:
An accuracy channel that maximizes task correctness.
An efficiency channel that enforces execution economy via conditional advantage estimation, applied only within accurate trajectories.
Cognitive curriculum: The decoupled architecture naturally forces the agent to first master task-solving capability before refining its self-reliance (reduced tool dependence).

Results

Extensive evaluations show that the final model, Metis, reduces tool invocations by several orders of magnitude while simultaneously improving reasoning accuracy.

--- *Auto-collected on 2026-04-12*