DeepSeek-R1 can solve International Mathematical Olympiad problems, yet it cannot book a flight ticket.
In early 2025, this contrast went viral. Reasoning models were dominating math benchmarks, while AI Agents stumbled in the real world—misclicking buttons, garbling form entries, and forgetting the first step by step three of a multi-step task.
You could blame "the model isn't big enough." But that explanation is too comfortable—it may have fooled you already. A survey published in April 2026 by independent researcher Chenchen Zhang poured cold water on it: he cataloged 47 credit assignment methods from 2024 to early 2026, and the resulting map shows Reasoning RL's half glowing with mature green light while Agentic RL's half remains nearly dark.
> Note: Credit Assignment — In RL, a model usually receives only a sparse final reward (task success/failure). Credit assignment is the task of decomposing that final reward onto each intermediate action, so the model knows what to change—like a student who only ever sees a total exam score, never an itemized error analysis.
One Map, Two Worlds
Zhang's core contribution is a two-dimensional taxonomy: granularity (Token → Segment → Step → Turn → Multi-Agent) crossed with methodology (Monte Carlo, Temporal Difference, Model-based, Game-theoretic, Info-theoretic). Representative methods include VinePPO (token-level MC), SPO/SCAR (segment-level), PURE, HICRA and PRM-series (step-level), ArCHer, AgentPRM, C3, CCPO, CAPO (turn-level), and CARL (multi-agent).
The most striking feature is the diagonal break from upper-left to lower-right:
| Dimension | Reasoning RL | Agentic RL | |:---|:---|:---| | Task | Math, code, logic | Operating computers, phones, web | | Length | Single generation, 500–30,000 tokens | Multi-turn, 100,000–1,000,000 tokens | | Environment | Deterministic | Stochastic | | Verifiability | Intermediate steps checkable | Intermediate states hard to judge | | Credit assignment | 🟢 Mature | 🔴 Frontier |
In Agentic settings, the same click may open a dialog or throw an error; the model cannot observe backend state; and "clicked the settings menu" cannot be judged correct or wrong in isolation. As Zhang puts it (with strong empirical support, [SE]): "From single-generation trajectories to multi-turn agent interactions, credit assignment shifts from an optimization convenience to a training necessity."
The GRPO Blind Spot
GRPO (Group Relative Policy Optimization), the algorithm behind DeepSeek-R1, elegantly handles reasoning credit assignment by comparing multiple candidate answers within a group—no separate Critic needed. But it relies on a hidden premise: final outcomes of multiple attempts at the same problem are comparable. In math, that holds. In agentic tasks—where one attempt stalls at the login page and another reaches the payment page, with diverging paths from step 5 onward—it does not. Zhang's meta-comparison of the GRPO family finds it robust for Reasoning but sharply degraded in Agentic settings with long horizons, stochastic transitions, and partial observability.
LLM-as-Critic: A Uniquely Modern Paradigm
Classical RL critics output scalars over numeric state vectors. But an LLM agent's intermediate state is a screenshot, HTML, a popup—states whose quality is best described in natural language. Hence the LLM-as-Critic paradigm (CAPO, SWEET-RL, LaRe, HCAPO, CriticSearch): pause mid-trajectory and let another LLM (or another set of weights) critique the step—"correctly opened the settings menu, but the Bluetooth option requires scrolling further down." Zhang tags this [LS]: limited but suggestive evidence, promising but not yet proven superior to value-function-based methods.
The Uncomfortable Inference
The survey implies: we may already be near the ceiling of Reasoning capability. The toolbox—PRM, VinePPO, GRPO—is largely complete; what remains is engineering. The real bottleneck is whether models can learn to *act* in the real world through trial and error. Notably, in a single week of March 2026, three independent counterfactual credit assignment papers appeared simultaneously—a sign of an accelerating field.
The Author's Falsifiable Bet
Agentic credit assignment will be the most important AI research direction of 2026–2027, and its breakthrough will decide which labs build genuinely usable AI Agents.
This is not a safe "worth watching" claim. If, in a year, a 100B-parameter model reaches 95% success on agentic tasks via simple scaling alone—no clever credit assignment needed—the bet is falsified. But if the best agentic systems come from teams doing fine-grained credit assignment on smaller models (ArCHer, CARL, C3), it pays off.
Without solving agentic credit assignment, the "AI Agents will change the world" narrative risks becoming a cargo cult: bamboo control towers, coconut-shell headsets—and no plane arriving. Until we figure out, across hundred-turn interaction chains, who gets the credit.
Paper Details
- Title: From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models
- Author: Chenchen Zhang (Independent Researcher)
- arXiv: 2604.09459
- Released: April 10, 2026 (v1); April 13, 2026 (v2)
- Category: cs.CL
- GitHub: Awesome-Credit-Assignment-in-LLM-RL
- Core contributions:
- Systematic survey of 47 credit assignment methods (41 core + 6 auxiliary), 2024–early 2026
- Two-dimensional taxonomy by granularity and methodology
- Explicit Reasoning-to-Agentic paradigm shift analysis
- Reusable community resources: structured paper database, reporting checklist, benchmark protocol
- Key claims with evidence levels:
[SE] Credit assignment is the core challenge of LLM RL; its importance grows with agenticization
2. [SE] Reasoning credit assignment is mature (token/segment/step-level methods effective)
3. [LS] Agentic credit assignment remains nascent (stochastic, partially observable, ultra-long-horizon challenges)
4. [LS] LLM-as-Critic is a paradigm unique to the LLM era with no direct classical RL counterpart
5. [AS] The field is accelerating (three counterfactual CA papers in one week, March 2026)