DeepSeek-R1 can solve International Math Olympiad problems but cannot book a plane ticket. This contrast—between reasoning models dominating benchmarks and AI agents fumbling in the real world—motivates a survey by independent researcher Chenchen Zhang: From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models (arXiv:2604.09459, April 2026).
The Core Argument
The problem is not that models are too small—it is that training signals are too coarse. The survey reviews 47 credit assignment methods (41 core + 6 auxiliary) from 2024 to early 2026 and maps a clear divide:
- Reasoning RL: credit assignment at token, segment, and step granularity is mature (VinePPO, PURE, PRM families, GRPO variants), enabled by deterministic environments and verifiable intermediate steps over single generations of 500–30,000+ tokens.
- Agentic RL: multi-turn interactions easily exceed 100 turns and 100,000–1,000,000 tokens, with stochastic transitions, partial observability, and unverifiable intermediate states. Here the field is still largely dark.
- Token/Step level (Reasoning): VinePPO, PURE, HICRA, PRM series
- Segment level: SPO, SCAR, Shapley-value methods
- Turn level (Agentic): AgentPRM, ArCHer, C3, CCPO, CAPO
- Multi-Agent: CARL, game-theoretic allocation
- Title: From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models
- Author: Chenchen Zhang (Independent Researcher)
- arXiv: 2604.09459 (v1: 2026-04-10, v2: 2026-04-13)
- Category: cs.CL
- Resource: Awesome-Credit-Assignment-in-LLM-RL
The author's strong-evidence conclusion: moving from single-generation trajectories to multi-turn agent interactions turns credit assignment from an optimization convenience into a training necessity.
A Two-Dimensional Taxonomy
Methods are organized by granularity (Token → Segment → Step → Turn → Multi-Agent) and methodology (Monte Carlo, Temporal Difference, Model-based, Game-theoretic, Information-theoretic). Notable entries:
The sharpest observation is the diagonal break: the mature upper-left (reasoning) and the primitive lower-right (agentic) regions of the map.
Why GRPO Breaks Down in Agentic Settings
GRPO succeeds by comparing multiple answers to the *same* problem group-wise, requiring no critic model. This works when final outcomes are comparable—as with math problems. In agentic scenarios, two attempts may diverge at step 5 and become incomparable thereafter; stochastic environments and partial observability destroy the comparability assumption. The survey's meta-comparison finds GRPO variants perform robustly for reasoning but degrade sharply on long-horizon agentic tasks.
LLM-as-Critic: A Uniquely LLM-Era Paradigm
Instead of numeric state vectors and scalar critics, LLM-as-Critic methods (CAPO, SWEET-RL, LaRe, HCAPO, CriticSearch) let another LLM evaluate intermediate steps in natural language—e.g., "the step correctly opened the settings menu, but you still need to scroll down for Bluetooth." This has no direct analogue in classical RL and is tagged [LS] limited but suggestive evidence—promising but unproven against value-function baselines.
The Uncomfortable Implication and the Bet
The survey suggests we may be nearing the ceiling of reasoning capability: the credit assignment toolbox there is already sufficient, with only engineering refinement left. The real bottleneck is whether models can learn to act in the real world through trial and error—requiring credit assignment precise enough to judge whether step 37 of a 100-turn interaction was a masterpiece or a disaster.
The author makes a falsifiable bet: agentic credit assignment will be the most important AI research direction of 2026–2027, deciding which labs build genuinely usable agents. If simple scaling alone gets agents to 95% success, the bet fails; if fine-grained credit assignment designs (ArCHer, CARL, C3) on smaller models win, it pays off. A signal of acceleration: in a single week of March 2026, three independent counterfactual credit assignment papers appeared.