ToolRL: Reward Is All Tool Learning Needs — A Systematic Analysis of Reward Design for Tool-Integrated Reasoning

> In April 2025, a UIUC team released ToolRL, the first systematic study of reinforcement learning reward design for tool selection and application tasks. The work shows that RL recipes from mathematical reasoning transfer poorly to tool learning, and through extensive ablations across four dimensions (type, scale, granularity, dynamics), proposes a fine-grained reward design that yields significant gains on multiple tool-use benchmarks, plus emergent behaviors such as tool refusal and intent clarification.

• Paper: ToolRL: Reward Is All Tool Learning Needs (arXiv: 2504.13958)
• Authors: Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, Heng Ji (University of Illinois Urbana-Champaign)
• Code: https://github.com/qiancheng0/ToolRL

Key points

• Paradigm gap: Unlike math reasoning (single correct answer, text-only derivation), tool-integrated reasoning (TIR) involves structured multi-layer correctness, tool calls plus environment feedback, and task-dependent optimal trajectory lengths. Binary correctness + length incentives produce sparse or misleading learning signals.
• SFT bottleneck: Supervised fine-tuning causes pattern memorization and poor generalization. On Bamboogle, SFT *degraded* Qwen2.5-7B accuracy from 69.6% to as low as 28.8%, motivating RL-based training.

The four dimensions of reward design

1. Reward type

• Format reward: checks structural conventions (<think>, <tool_call>, <response> tokens in correct order).
• Correctness reward: decomposed into tool-name matching (high weight), parameter-name matching, and parameter-value matching.
• Rationale: tool-call errors are often local (e.g., wrong parameter value but right tool). Fine-grained decomposition gives learning signal from partial correctness instead of all-or-nothing sparse reward.

2. Reward scale

| Config | Qwen2.5-1.5B | Qwen2.5-3B | Finding | |---|---|---|---| | Original (correctness-dominant) | 46.20% | 52.98% | Best baseline | | Equal Max | 39.47% | 51.76% | Worse | | Two Stage (coarse switch) | 38.85% | 50.66% | Harmful | | Dynamic (smooth gradient) | 45.71% | 53.81% | Best smooth transition |

Correctness reward should always dominate. Coarse two-stage switching (format first, then correctness) disrupts training dynamics; smooth gradient transitions work better.

3. Reward granularity

| Granularity | Qwen2.5-1.5B | |---|---| | Original (tool name / param name / param value matched separately) | 46.20% | | Fine-grained | 40.71% | | Intermediate | 37.65% | | Coarse (whole tool call exact match) | 36.72% |

Finer granularity yields higher performance. Coarse rewards are too sparse, making credit assignment difficult.

4. Reward dynamics (length rewards)

| Length reward | Qwen2.5-1.5B | Qwen2.5-3B | |---|---|---| | None | 46.20% | 52.98% | | Fixed length reward | 33.23% (harmful) | 48.89% | | Dynamic length reward | 28.51% (very harmful) | 48.24% |

Counter-intuitive finding: length rewards that encourage longer reasoning hurt tool learning. Tool use requires precise decision-making, not extended deliberation—over-thinking can delay actions and mistime tool calls.

Experimental results: GRPO cold start wins

BFCL (Berkeley Function Call Leaderboard, Qwen2.5-7B)

| Method | Overall | AST | Exec | Live | Multi-Turn | |---|---|---|---|---|---| | Raw | 41.97% | 66.02% | 70.11% | 53.51% | 4.25% | | SFT400 | 34.08% | 69.29% | 66.68% | 41.40% | 0.00% | | SFT4k | 36.53% | 45.15% | 53.50% | 57.13% | 0.75% | | PPO Cold Start | 46.68% | 79.33% | 78.16% | 63.17% | 0.38% | | GRPO Cold Start | 58.38% | 86.17% | 78.25% | 74.90% | 18.12% |

GRPO Cold Start beats all baselines across the board, especially in multi-turn interaction (18.12% vs 4.25%), suggesting GRPO's group-normalization suits multi-step decision tasks.

SFT init vs cold start

SFT-initialized models earn higher training rewards but generalize *worse* than cold-start models—higher training reward does not equal better generalization. Direct RL from instruct models avoids the SFT memorization trap.

Bamboogle (free-form tool use)

| Method | Accuracy | Avg tool calls | |---|---|---| | Raw | 69.6% | 1.42 | | SFT400 | 28.8% | 3.71 | | SFT4k | 30.4% | 1.06 | | PPO Cold Start | 48.0% | 1.25 | | GRPO Cold Start | 72.0% | 1.63 |

Emergent behaviors

• Active tool refusal: facing irrelevant tools, the model identifies task–tool mismatch and answers directly. Example: given a "find critical points" tool for a slope question, it answers -1/3 via the negative-reciprocal rule instead of calling the tool.
• Intent clarification: with insufficient user information (e.g., buying movie tickets without a movie name or date), the model asks for missing details rather than making a doomed call.
• Neither behavior was explicitly trained; both emerge indirectly from fine-grained correctness and format rewards.

Design principles

1. Granularity first: fine-grained decomposition beats coarse holistic judgment. 2. Correctness dominance: correctness reward weight should exceed format reward. 3. Gradients over stage switches: reward scales should transition smoothly. 4. Task-fit validation: cross-domain tricks like length rewards must be validated per task. 5. Cold start works: RL directly from instruct models avoids SFT memorization.

Conclusion

ToolRL provides the first systematic reward-design guide for RL-based tool learning. Its deeper message: tool use is not merely "calling APIs" but a core manifestation of decision intelligence—knowing when to act, refuse, or clarify. Carefully designed rewards can induce these metacognitive capabilities autonomously, laying groundwork for more autonomous LLM agents.