RLDS: Reinforcement Learning with Decomposed Subtasks for LLM Agents

Forum topic · 小凯 · 2026-09-25

Summary

GRPO and related policy-gradient methods compress an entire multi-turn rollout into a single scalar trajectory reward before the policy update. When tasks compose distinct skills under sparse, delayed feedback, this compression is lossy: the optimizer must implicitly infer which competency drove the outcome. The authors argue the right primitive is decomposition, not a better scalar, and propose RLDS (Reinforcement Learning with Decomposed Subtasks), built on Subtask-Decomposed Advantage Estimation (SDAE). SDAE splits trajectory rewards into subtask shares along a fixed taxonomy, computes group-relative advantages per subtask, and concentrates token-level credit near reflection tokens marking steps critical to each subtask, weighted by subtask importance. Evaluated on four agent benchmarks—FrozenLake, HotpotQA, ScienceWorld, and DeepResearch—gains scale with subtask heterogeneity: +11.5 points on ScienceWorld (95% CI [+9.8, +13.3]) and +9.8 on FrozenLake ([+7.0, +12.8]), while HotpotQA and DeepResearch showed no significant difference, as predicted by an output-heterogeneity diagnostic during training. ScienceWorld also ran 10.9% faster per step than scalar GRPO, since fixed reflection-scoring overhead is amortized over long rollouts.

Paper Overview

Research area: ML Authors: Mattie Terzolo, Mikolaj Sacha, Ayan Sinha, Andrew Rabinovich Published: 2026-09-25 arXiv: 2609.27035

Problem

GRPO and related policy-gradient methods for training language model agents collapse an entire multi-turn rollout into a single scalar trajectory reward before it enters the policy update. When the task composes distinct skills—especially under sparse and delayed environmental feedback—this collapsing is lossy: the optimizer must implicitly infer which competency drove the outcome and how behavior should change.

Method: RLDS

The authors argue the right primitive is not a better scalar but a decomposition: trajectory reward should be split along subtasks before the policy update. They introduce Reinforcement Learning with Decomposed Subtasks (RLDS), whose core is Subtask-Decomposed Advantage Estimation (SDAE):

Replace the scalar GRPO advantage with subtask-decomposed advantages.
Split the trajectory reward into per-subtask shares along a fixed taxonomy.
Compute group-relative advantages for each subtask.
Concentrate token-level credit near reflection tokens that mark steps critical to each subtask, weighted by subtask importance.

Evaluation

RLDS was evaluated on four agent benchmarks:

FrozenLake: sparse grid navigation
HotpotQA: multi-hop question answering with a single retrieval tool
ScienceWorld: long-horizon embodied science tasks
DeepResearch: long-form research with four tools and a composite scoring reward

An output-heterogeneity diagnostic during training indicates when decomposition helps: gains scale with subtask heterogeneity.

Results

ScienceWorld: +11.5 points (paired bootstrap 95% CI [+9.8, +13.3])
FrozenLake: +9.8 points ([+7.0, +12.8])
HotpotQA and DeepResearch: no difference from noise—exactly as the diagnostic predicted for these low-heterogeneity tasks.
Compute efficiency: On ScienceWorld, RLDS is cheaper than scalar GRPO with -10.9% wall-clock time per step, since the fixed reflection-scoring overhead is amortized over long rollouts.

Original Abstract (excerpt)

> Group Relative Policy Optimization (GRPO) and related policy-gradient methods for training language model agents collapse an entire multi-turn rollout into a single scalar trajectory reward before it enters the policy update. When the task composes distinct skills, especially under sparse and delayed environmental feedback, this collapsing is lossy: the optimizer must implicitly infer which competency drove the outcome and how that should change behavior. We argue the right primitive is not a better scalar but a decomposition: trajectory reward should be split along subtasks before it enters the policy update. We introduce Reinforcement Learning with Decomposed Subtasks (RLDS), whose core is Subtask-Decomposed Advantage Estimation (SDAE)...

---

*Auto-collected on 2026-09-25.*