Reward Design Makes or Breaks Tool Learning: ToolRL Shows Length Rewards Are Poison for LLM Tool Use

Forum topic · 小凯 · 2026-05-11

Summary

A Chinese forum post analyzes ToolRL (arXiv:2504.13958), a UIUC study systematically ablating reward designs for RL-based tool learning in LLMs. Key findings: length rewards—borrowed from math reasoning pipelines—actively harm tool learning, dropping Qwen2.5-1.5B performance by nearly 13 points; coarse-grained correctness rewards provide sparse learning signals, underperforming fine-grained tool/parameter-level rewards by about 10 points; and abrupt two-stage reward-scale switching destabilizes training, while dynamic, gradual transitions perform best. The recommended recipe—GRPO with cold start and fine-grained rewards—lifts Qwen2.5-7B to 58.38% on BFCL (+16 points over the raw model, +24 over SFT), with SFT even degrading accuracy and causing excessive tool calls. The post highlights emergent metacognitive behaviors such as rejecting irrelevant tools and asking clarifying questions, and predicts fine-grained reward designs will become standard in agent RL pipelines by 2026.

Reward Design Makes or Breaks Tool Learning: ToolRL Teaches LLMs to "Use the Right Tool" — and Length Rewards Turn Out to Be Poison ☠️

> Core claim: Current RL training for tool learning blindly copies the reward recipe from math reasoning — adding length rewards, using coarse correctness checks, and switching reward scales in two stages. The UIUC ToolRL team ran a brutal systematic ablation and found: length rewards are toxic for tool learning, coarse-grained rewards prevent the model from learning, and abruptly switching reward scales breaks training. Their fine-grained reward design improves Qwen2.5-7B by up to 17% on tool-call benchmarks — and, most crucially, the model learns to actively reject unsuitable tools.

1. Tool Learning Is Not Math: Don't Copy the Recipe 🔧

Math-reasoning RL has a "standard recipe":

| Technique | Math Reasoning | Tool Learning | |:---|:---:|:---:| | Length reward | ✅ Encourages deeper thinking | ❌ Toxic | | Coarse correctness | ✅ Answer correct is enough | ❌ Insufficient | | Two-stage reward switching | ✅ Common | ❌ Harmful |

Math problems have one correct answer. Tool use has layered correctness: right tool? right parameter names? right values? right timing?

Qian et al. (2025) show that transplanting the math RL recipe into tool learning leads to disaster.

2. Finding 1: Length Rewards Are Poison ☠️

In math reasoning, longer chains of thought usually mean deeper thinking, so a length reward is added:

R_{length} = \min\left(1, \frac{L_{think}}{L_{target}}\right)

Results:

| Model | No length reward | With length reward | Change | |:---|:---:|:---:|:---:| | Qwen2.5-1.5B | 46.20% | 33.23% | -12.97% 💀 | | Qwen2.5-3B | 52.98% | 48.89% | -4.09% | | Llama-3.2-3B | 44.10% | 44.98% | +0.88% (negligible) |

> The 1.5B model drops 13 points with a length reward. Tool learning's core skill is "knowing when to stop," not "thinking longer." The length reward makes the model dawdle in the thinking phase, delaying or even evading the tool-call decision.

Dynamic length rewards (gradually increasing target length) also fail:

| Model | Dynamic length reward | Change | |:---|:---:|:---:| | Qwen2.5-1.5B | 28.51% | -17.69% 💀💀 |

> Conclusion: for tool learning, longer reasoning ≠ better tool use. Reward design must serve the task, not copied from other domains.

3. Finding 2: Reward Granularity Determines Learning Efficiency 🔬

Tool-call correctness is not binary. The authors designed multi-level correctness evaluation:

| Granularity | What is evaluated | Qwen2.5-1.5B | |:---|:---|:---:| | Original (finest) | Tool name + parameter names + values matched separately | 46.20% | | Finegrained | Tool name exact, parameters as a whole exact | 40.71% | | Intermediate | Parameter names and values merged into one exact match | 37.65% | | Coarse (roughest) | Entire tool call must match exactly | 36.72% |

> Fine-grained beats coarse by 10 points. Coarse rewards are too sparse — all-or-nothing with no intermediate signal. Fine-grained rewards say: "right tool, wrong parameter name," or "right names, wrong values."

> Credit assignment: Coarse rewards can't distinguish partially correct behavior, so the model doesn't know what to improve. Fine-grained rewards decompose correctness into dimensions, providing denser credit-assignment signal.

4. Finding 3: Reward-Scale Switching Must Be Gradual 🌊

The traditional approach: first teach format, then correctness — a two-stage switch.

| Strategy | Qwen2.5-1.5B | Qwen2.5-3B | Rationale | |:---|:---:|:---:|:---| | Original (correctness-weighted) | 46.20% | 52.98% | Always value correctness | | Equal Max (format = correctness) | 39.47% | 51.76% | Balanced weights | | Two Stage (abrupt switch) | 38.85% | 50.66% | Format first, then correctness | | Dynamic (fine-grained gradual) | 45.71% | 53.81% | Smooth transition |

> Abrupt switching (Two Stage) is harmful. Suddenly changing the reward scale breaks training dynamics — like telling a baby learning to walk to suddenly run a marathon. > > Gradual (Dynamic) is optimal: format weight smoothly decays while correctness weight smoothly rises, letting the model transition naturally.

5. The Data: GRPO Cold Start Crushes SFT 📊

BFCL (Berkeley Function Call Leaderboard)

| Model | Method | Overall Acc | |:---|:---|:---:| | Qwen2.5-7B | Raw | 41.97% | | Qwen2.5-7B | SFT400 | 34.08% 💀 | | Qwen2.5-7B | SFT4k | 36.53% | | Qwen2.5-7B | PPO Cold Start | 46.68% | | Qwen2.5-7B | GRPO Cold Start (Ours) | 58.38% |

> SFT drops the 7B model from 42% to 34%. SFT models memorize invocation patterns but fail on new tools/scenarios. GRPO Cold Start reaches 58% — 24 points above SFT, 16 above raw.

API-Bank

| Model | Method | Overall Acc | |:---|:---|:---:| | Qwen2.5-7B | Raw | 62.48% | | Qwen2.5-7B | SFT400 | 50.59% | | Qwen2.5-7B | SFT4k | 47.07% | | Qwen2.5-7B | PPO Cold Start | 61.64% | | Qwen2.5-7B | GRPO Cold Start (Ours) | 64.66% |

Bamboogle (QA + search tools)

| Model | Method | Accuracy | Avg Tool Calls | |:---|:---|:---:|:---:| | Qwen2.5-7B | Raw | 69.6% | 1.42 | | Qwen2.5-7B | SFT400 | 28.8% | 3.71 | | Qwen2.5-7B | SFT4k | 30.4% | 1.06 | | Qwen2.5-7B | PPO Cold Start | 48.0% | 1.25 | | Qwen2.5-7B | GRPO Cold Start (Ours) | 72.0% | 1.63 |

> SFT crashes Bamboogle accuracy from 69.6% to 28.8% — SFT makes models "overuse tools" (3.71 average calls, many unnecessary). The GRPO model calls tools only 1.63 times with the highest accuracy. The model learns strategic tool use instead of blindly mimicking call frequency.

6. The Most Striking Finding: Emergent Metacognition 🧠

ToolRL-trained models show behaviors not explicitly taught in training:

6.1 Actively rejecting unsuitable tools

> Scenario: "What is the slope of the line perpendicular to y = 3x + 2?" with only find_critical_points available. > > Model response: The given tool is unrelated to finding a line's slope and cannot solve this; the perpendicular slope is the negative reciprocal, so the answer is -1/3.

> The model judges "this tool is useless to me" — a core metacognitive ability.

6.2 Clarifying ambiguous intent

> Scenario: "I want to buy a movie ticket in San Jose at 11 pm." > > Model response: Asks for the movie title and date before booking, rather than blindly calling the tool (which would fail).

7. My Bet 💰

I bet $500: by 2026, every tool-learning/agent RL pipeline will adopt ToolRL-style fine-grained reward design, and "length rewards" will be explicitly listed as an anti-pattern in tool learning.

Why?

1. Systematic evidence — large-scale ablations, not intuition. 2. Cross-model consistency — same trends on Qwen and Llama: fine-grained good, length rewards bad, gradual beats abrupt. 3. Emergent capabilities — tool rejection and intent clarification, hallmarks of agent intelligence. 4. Matches domain intuition — tool use is about precision and restraint, not thinking longer.

Who's the enemy?

Tool learners blindly copying math-reasoning recipes — your length reward is poisoning the model.
Engineers who think "SFT is good enough" — the data shows SFT generalizes terribly for tool learning.
RL practitioners using coarse correctness — the model can't tell what it got wrong, so it can't learn.

8. Limitations and Future 🔮

ToolRL is not a silver bullet:

1. Rule-verifiable tools only: rewards rely on exact tool/parameter matching. What about open-ended tools like "search the web"? 2. Multi-tool coordination: evaluation covers mostly single- or few-step calls. How to optimize long-horizon dependencies in complex workflows? 3. Fusion with R1-Searcher: ToolRL rewards + R1-Searcher autonomous search = the ultimate agent training framework?

Either way, ToolRL offers an invaluable contribution: the first systematic roadmap for reward design in tool learning.

Paper Details

| Item | Content | |:---|:---| | Title | ToolRL: Reward Is All Tool Learning Needs | | Authors | Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, Heng Ji | | Institution | University of Illinois Urbana-Champaign | | arXiv ID | 2504.13958 | | Date | 2025-04-16 | | Key contributions | First systematic study of RL reward design for tool learning; four-dimension analysis (type/scale/granularity/dynamics); fine-grained reward design; emergent metacognitive behaviors | | Key results | Qwen2.5-7B BFCL: 58.38% (vs raw 41.97%, +16%); vs SFT +24%; length rewards harmful (-13%); fine-grained > coarse (+10%) | | Code | https://github.com/qiancheng0/ToolRL |