Reward Design Makes or Breaks Tool Learning: ToolRL Teaches LLMs to Use Tools Right—and Length Rewards Are Poison

Forum topic · 小凯 · 2026-05-11

Summary

ToolRL, a study from UIUC (arXiv:2504.13958), systematically analyzes reward design for reinforcement learning in tool learning, showing that blindly importing math-reasoning reward recipes harms performance. Key findings: length rewards are toxic—Qwen2.5-1.5B dropped 12.97 points on tool-calling benchmarks; coarse-grained correctness rewards underperform fine-grained rewards by roughly 10 points because they provide no credit assignment signal; abrupt two-stage reward scale switching destabilizes training, while gradual dynamic scaling works best. With fine-grained rewards and GRPO cold-start training, Qwen2.5-7B reaches 58.38% on the Berkeley Function Call Leaderboard (vs 41.97% raw and below 37% for SFT variants) and 72.0% on Bamboogle. Notably, trained models show emergent meta-cognitive behaviors such as proactively rejecting irrelevant tools and asking clarifying questions. The authors conclude that reward design must serve the task itself, positioning ToolRL as the first systematic roadmap for tool-learning reward engineering.

Reward Design Makes or Breaks Tool Learning: ToolRL Teaches LLMs to 'Use Tools Right'—and Length Rewards Are Poison ☠️

> Core claim: Most current RL training for tool learning blindly copies reward recipes from math reasoning—adding length rewards, using coarse correctness checks, and switching reward scales abruptly. The UIUC ToolRL team ran a brutal systematic ablation and found that length rewards are toxic for tool learning, coarse rewards prevent the model from learning, and sudden reward-scale switches break training. Their fine-grained reward design boosts Qwen2.5-7B by up to 17% on tool-calling benchmarks—and, most critically, the model learns to proactively refuse unsuitable tools.

---

1. Tool learning is not math—stop copying the recipe 🔧

Math-reasoning RL has a 'standard recipe':

| Technique | Math reasoning | Tool learning | |:---|:---:|:---:| | Length reward | ✅ Encourages deeper thinking | ❌ Toxic | | Coarse correctness | ✅ Answer correctness suffices | ❌ Insufficient | | Two-stage reward switch | ✅ Common | ❌ Harmful |

> Why doesn't the math recipe transfer? Math problems have one correct answer. Tool use has multi-level correctness: Did the model pick the right tool? Are parameter names right? Are values right? Is the ordering right?

Qian et al. (2025) show that directly porting math RL recipes to tool learning is a disaster.

---

2. Finding 1: Length rewards are poison ☠️

In math reasoning, longer chains of thought usually mean deeper thinking, so researchers add a length reward:

R_{length} = \min\left(1, \frac{L_{think}}{L_{target}}\right)

Result?

| Model | No length reward | With length reward | Change | |:---|:---:|:---:|:---:| | Qwen2.5-1.5B | 46.20% | 33.23% | -12.97% 💀 | | Qwen2.5-3B | 52.98% | 48.89% | -4.09% | | Llama-3.2-3B | 44.10% | 44.98% | +0.88% (negligible) |

> The 1.5B model lost 13 points with a length reward. Tool learning's core skill is 'knowing when to stop,' not 'thinking longer.' Length rewards make models dawdle in the thinking phase, delaying or even avoiding tool calls.

Dynamic length rewards (gradually increasing target length) are just as bad:

| Model | Dynamic length reward | Change | |:---|:---:|:---:| | Qwen2.5-1.5B | 28.51% | -17.69% 💀💀 |

> Conclusion: for tool learning, longer reasoning ≠ better tool use. Reward design must serve the task itself, not borrowed experience.

---

3. Finding 2: Reward granularity determines learning efficiency 🔬

Tool-call correctness is not binary. The authors design three levels of correctness evaluation:

| Granularity | What's evaluated | Qwen2.5-1.5B | |:---|:---|:---:| | Original (finest) | Tool name + parameter names + values matched separately | 46.20% | | Finegrained | Tool name exact, parameters matched as a whole | 40.71% | | Intermediate | Parameter names and values as a single exact match | 37.65% | | Coarse (roughest) | Entire tool call must match exactly | 36.72% |

> Fine-grained rewards beat coarse rewards by 10 points. Coarse rewards are too sparse—all-or-nothing, with no learning signal in between. Fine-grained rewards tell the model: 'right tool, wrong parameter name' or 'right name, wrong value.'

> Credit assignment: In RL, credit assignment means determining which behaviors led to the final reward. Coarse rewards can't distinguish partially correct behavior; fine-grained rewards decompose correctness into dimensions and provide a denser signal.

---

4. Finding 3: Reward-scale switching must be gradual 🌊

Traditional approach: learn format first, then correctness—a two-stage switch. ToolRL tests three scale strategies:

| Strategy | Qwen2.5-1.5B | Qwen2.5-3B | Principle | |:---|:---:|:---:|:---| | Original (correctness-weighted) | 46.20% | 52.98% | Always weight correctness | | Equal Max (format = correctness) | 39.47% | 51.76% | Balanced weights | | Two Stage (abrupt switch) | 38.85% | 50.66% | Format first, then correctness | | Dynamic (gradual) | 45.71% | 53.81% | Smooth transition |

> Abrupt switching (Two Stage) is harmful. Suddenly changing reward scales breaks training dynamics—the model learns format in 30 steps, then is told format no longer matters. > Gradual (Dynamic) is best. Smoothly decaying format weight and rising correctness weight lets the model transition naturally.

---

5. The data: GRPO Cold Start crushes SFT 📊

BFCL (Berkeley Function Call Leaderboard)

| Model | Method | Overall Acc | |:---|:---|:---:| | Qwen2.5-7B | Raw | 41.97% | | Qwen2.5-7B | SFT400 | 34.08% 💀 | | Qwen2.5-7B | SFT4k | 36.53% | | Qwen2.5-7B | PPO Cold Start | 46.68% | | Qwen2.5-7B | GRPO Cold Start (Ours) | 58.38% |

> SFT dropped the 7B model from 42% to 34%. SFT models memorize call patterns from training data and fail on new tools or scenarios. GRPO Cold Start from scratch hits 58%—24 points above SFT, 16 points above raw.

API-Bank

| Model | Method | Overall Acc | |:---|:---|:---:| | Qwen2.5-7B | Raw | 62.48% | | Qwen2.5-7B | SFT400 | 50.59% | | Qwen2.5-7B | SFT4k | 47.07% | | Qwen2.5-7B | PPO Cold Start | 61.64% | | Qwen2.5-7B | GRPO Cold Start (Ours) | 64.66% |

Bamboogle (QA + search tools)

| Model | Method | Accuracy | Avg Tool Calls | |:---|:---|:---:|:---:| | Qwen2.5-7B | Raw | 69.6% | 1.42 | | Qwen2.5-7B | SFT400 | 28.8% | 3.71 | | Qwen2.5-7B | SFT4k | 30.4% | 1.06 | | Qwen2.5-7B | PPO Cold Start | 48.0% | 1.25 | | Qwen2.5-7B | GRPO Cold Start (Ours) | 72.0% | 1.63 |

> SFT collapsed Bamboogle accuracy from 69.6% to 28.8% by making the model overuse tools (3.71 average calls). The GRPO model makes 1.63 calls with the highest accuracy—it learned strategic tool use rather than blindly imitating call frequency.

---

6. The most surprising finding: emergent meta-cognition 🧠

ToolRL-trained models show emergent behaviors not explicitly taught:

6.1 Proactively rejecting unsuitable tools

> Scenario: 'What is the slope of the line perpendicular to y = 3x + 2?' > Available tool: find_critical_points > Model: 'The given tool find_critical_points is unrelated to finding a perpendicular slope and cannot solve this. The perpendicular slope is the negative reciprocal: -1/3.'

> The model learned to judge 'this tool is useless for me'—a core meta-cognitive skill.

6.2 Clarifying ambiguous intent

> Scenario: 'I want to buy a movie ticket in San Jose at 11 PM' > Model: 'To purchase the ticket, I need the movie title and the specific date. Please provide them.'

> Instead of blindly calling a booking tool (which would fail), the model asks for missing information.

---

7. My bet 💰

I bet $500: by 2026, all tool-learning/agent RL pipelines will adopt ToolRL-style fine-grained reward design, and length rewards will be explicitly listed as an anti-pattern in tool learning.

Why?

1. Systematic evidence: Large-scale ablations, not intuition, prove what helps and what harms. 2. Cross-model consistency: Same trends on Qwen and Llama—fine-grained good, length rewards bad, gradual beats abrupt. 3. Emergent capabilities: Proactive refusal and intent clarification—hallmarks of agent intelligence. 4. Domain intuition: Tool use rewards precision and restraint, not longer thinking.

Who is the enemy?

Tool learners blindly copying math-reasoning recipes—your length rewards are poisoning your model.
Engineers assuming 'SFT is good enough'—the data shows SFT generalizes poorly in tool learning.
RL practitioners using coarse correctness checks—your model doesn't know what it got wrong, so it can't learn.

---

8. Limitations and future 🔮

ToolRL is not a silver bullet:

1. Rule-verifiable tools only: Rewards rely on exact tool-name and parameter matching. How to design rewards for open-ended tools (e.g., 'search the web')? 2. Multi-tool coordination: Evaluation focuses on single-step or few-step calls. How to optimize long-horizon tool dependencies in complex workflows? 3. Fusion with R1-Searcher: ToolRL's reward design + R1-Searcher's autonomous search = the ultimate agent training framework?

Either way, ToolRL offers an invaluable contribution: the first systematic roadmap of reward design for tool learning.

---

Paper details

| Item | Content | |:---|:---| | Title | ToolRL: Reward Is All Tool Learning Needs | | Authors | Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, Heng Ji | | Institution | University of Illinois Urbana-Champaign | | arXiv ID | 2504.13958 | | Date | 2025-04-16 | | Key contributions | First systematic study of RL reward design for tool learning; four-axis analysis (type/scale/granularity/dynamics); fine-grained reward design; emergent meta-cognitive behaviors | | Key results | Qwen2.5-7B BFCL: 58.38% (vs Raw 41.97%, +16%); +24% vs SFT; length rewards harmful (-13%); fine-grained > coarse (+10%) | | Code | https://github.com/qiancheng0/ToolRL |