This article analyzes the TRACE framework from Tsinghua University and Tencent, which addresses a major inefficiency in agentic reinforcement learning with verifiable rewards (RLVR): roughly 65–73% of rollout samples contribute almost nothing to policy updates because their reward contrast is too low (all-pass or all-fail rollouts). TRACE reformulates prompt filtering, rollout count allocation, and prefix branching as a single tree-anchored budget allocation problem, solved in two stages using a shared success-probability predictor. The first stage allocates root-node budgets to prompts with high contrast potential; the second stage extends only prefixes whose subtrees still hold contrasting outcomes. Across math reasoning (Qwen3-8B/14B), multi-hop QA, and function calling, TRACE improves accuracy by 0.7–2.8 points at equal sampling cost and raises the effective-signal ratio from 26.8–34.7% (GRPO) to 59.7–60.6%, roughly doubling useful learning signal per unit compute. Three theoretical propositions back the framework.
TRACE: Tree-Based Rollout Budget Allocation Cuts Agent RL Training Waste by Half
Summary
This article analyzes the TRACE framework from Tsinghua University and Tencent, which addresses a major inefficiency in agentic reinforcement learning with verifiable rewards (RLVR): roughly 65–73% of rollout samples contribute almost nothing to policy updates because their reward contrast is too low (all-pass or all-fail rollouts). TRACE reformulates prompt filtering, rollout count allocation, and prefix branching as a single tree-anchored budget allocation problem, solved in two stages using a shared success-probability predictor. The first stage allocates root-node budgets to prompts with high contrast potential; the second stage extends only prefixes whose subtrees still hold contrasting outcomes. Across math reasoning (Qwen3-8B/14B), multi-hop QA, and function calling, TRACE improves accuracy by 0.7–2.8 points at equal sampling cost and raises the effective-signal ratio from 26.8–34.7% (GRPO) to 59.7–60.6%, roughly doubling useful learning signal per unit compute. Three theoretical propositions back the framework.
This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177981291