DeepMind's CSRO: LLMs Write Python as Readable Multi-Agent Game Policies

Forum topic · 小凯 · 2026-05-25

Summary

Google DeepMind researchers propose CSRO (Code-Space Response Oracles), a new take on the PSRO multi-agent reinforcement learning framework that replaces deep RL best-response oracles with LLM program synthesis. Instead of producing black-box neural network policies, CSRO prompts a large language model to generate commented, executable Python classes implementing an act(obs) method as the best response. The paper introduces three oracle variants: ZeroShot (one-shot code generation), LinearRefinement (regeneration on negative payoffs, up to 10 iterations), and AlphaEvolve (distributed evolutionary code search). On repeated Rock-Paper-Scissors and repeated Leduc Hold'em, CSRO variants substantially outperform PSRO-IMPALA baselines, with LinearRefinement matching a per-turn LLM agent while using roughly 50x fewer LLM calls, and AlphaEvolve reaching exploitability of 4.4 on Leduc—second only to CFR+. Generated policies are human-readable, revealing explicit opponent modeling and expected-value computation. CSRO reframes policy learning as code generation in an auditable program space, suited to high-stakes domains like finance bidding and safety games.

Paper: Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models Authors: Daniel Hennes¹*, Zun Li¹*, John Schultz¹, Marc Lanctot¹ (*Equal contributions) Institution: Google DeepMind Link: https://arxiv.org/abs/2603.10098

The Pain Point of PSRO: Equilibria You Can't Read

PSRO (Policy-Space Response Oracles, Lanctot et al., 2017) elegantly computes game equilibria by iteratively maintaining a policy population: each round it finds a best response to the current opponent mixture, adds it to the pool, and re-solves the equilibrium.

But the framework has a critical blind spot: best responses are computed by a deep RL oracle, producing black-box neural networks. You get an "optimal policy" with no idea why it picks an action. Debugging is guesswork; in high-stakes settings (financial bidding, security games, medical decisions), "performant but opaque" is a liability, not a feature.

The Core Shift: From Parameter Optimization to Program Synthesis

CSRO replaces the deep RL oracle with an LLM. Instead of fitting Q-values or policy distributions, the LLM directly writes a commented Python class implementing act(obs). A policy is an executable, annotated Python class.

This brings benefits beyond interpretability: 1. Naturally readable policies — comments, docstrings, and variable names state intent 2. Transparent opponent modeling — opponent policies are also code you can read 3. Composable, reusable — policies are importable modules, not weight matrices 4. Human knowledge injection — LLM pretraining knowledge directly shapes policy design

Three Oracle Variants

ZeroShot

One prompt, one complete policy. The prompt includes the game rules in natural language, a precise API spec, a description (or full source) of opponent policies, and instructions to produce the best response. ZeroShot reaches PopReturn 130.2 on RRPS and 40.4 on Leduc, far above PSRO-IMPALA.

LinearRefinement

If the generated policy scores a negative payoff, the error is fed back to the LLM for regeneration, up to M=10 iterations. Key results:

LinearRefinement(code, Top5) achieves AggScore 122.1 on RRPS, nearly matching a dedicated LLM agent (Gemma 3 27B called every turn, AggScore 126.0)
Against an AlwaysFold opponent on Leduc, payoff 57.3 vs PSRO-IMPALA's 40.7
CSRO uses 20–200 LLM calls per reusable policy, while baseline LLM agents need ~1000 calls per game — a 50x efficiency gain

AlphaEvolve

RRPS: mean exploitability 25.2, as low as 3.3
Leduc: exploitability 4.4, payoff 110.3 against AlwaysCall (PSRO-IMPALA: 57.7)

Experimental Results

Repeated Rock-Paper-Scissors (RRPS)

| Method | PopReturn ↑ | PopExpl ↓ | AggScore ↑ | |:---|:---|:---|:---| | PSRO-IMPALA | −108.9 | 423.2 | −532.1 | | ZeroShot | 130.2 | 66.7 | 63.5 | | LinearRefinement(code, Top5) | 159.8 | 37.7 | 122.1 | | AlphaEvolve | 50.5 | 25.2 | 25.4 | | Baseline LLM Agent (Gemma 3 27B) | 193.2 | 67.2 | 126.0 |

PSRO-IMPALA nearly collapses in opponent-modeling repeated play (AggScore −532.1); all CSRO variants are strongly positive.

Repeated Leduc Hold'em

| Method | PopReturn ↑ | PopExpl ↓ | AggScore ↑ | |:---|:---|:---|:---| | PSRO-IMPALA | 13.3 | 58.4 | −45.0 | | CFR+ | 39.8 | 0.0 | 39.8 | | ZeroShot | 40.4 | 19.6 | 20.7 | | LinearRefinement | 43.8 | 9.8 | 34.0 | | AlphaEvolve | 49.3 | 4.4 | 44.9 |

AlphaEvolve is second only to the theoretical CFR+ on exploitability (4.4 vs 0.0), without traversing the full game tree.

What Do the Generated Policies Look Like?

RRPS best policy: an ensemble of 32 predictors (higher-order Markov models, reactive models, heuristic detectors). Expert votes are weighted by the fifth power of their scores for decisive opponent modeling, including a second-order theory-of-mind component — inferring what model the opponent is using to predict itself, then exploiting it.

Leduc best policy: explicit expected-value computation. The agent estimates its equity (showdown win rate) and the opponent's folding probability, then weighs the two outcomes. In code you can see:

vs AlwaysCall (fold probability ≈ 0) → degrades to "pure value betting," raising only with strong hands
vs AlwaysFold (fold probability ≈ 100%) → becomes "pure bluffing," hand strength becomes irrelevant

Such transparent adaptation is impossible to observe directly in black-box networks.

Why Can LLMs Serve as Good Oracles?

1. Strategic transfer of pretraining knowledge — the LLM has read countless descriptions of rock-paper-scissors and poker strategy; CSRO *orchestrates existing knowledge in context* rather than training from scratch. 2. Opponent abstraction in context — for large policy pools, CSRO uses a Top-5 filter (highest equilibrium-probability opponents) or LLM-generated summaries. Ablations show exploitability explodes to 614.2 without opponent input. 3. Code as a policy representation — code offers structure, composition, and state management: internal variables for opponent statistics, helper functions for EV computation, sub-modules like the 32-predictor ensemble.

Limitations and Open Questions

1. LLM capability ceiling — CSRO is bound to the underlying model's code-generation and reasoning ability 2. API cost — 50x cheaper than per-turn agents, but still more expensive than RL training 3. Scalability — fitting huge state spaces (Stratego, StarCraft) and opponent policies into context remains open 4. Syntactic failure rate — error handling and regeneration logic are required; not every generation compiles

Still, the direction is clear: when the goal is *understandable* strong performance rather than raw performance, program-synthesis oracles are a natural evolution of PSRO.

Verdict: A Paradigm Shift from Parameter Space to Code Space

CSRO's real significance is not "LLMs replacing RL" but redefining the representation space of policies. Traditional multi-agent learning optimizes in parameter space — unreadable by design. CSRO moves policies into code space, where they are composable, commented, and human-auditable. An expert can read a CSRO policy and judge whether a bluffing routine is too aggressive for a high-risk setting — then edit the code directly instead of retraining a network. In finance, military simulation, and medical resource allocation, auditability often matters more than optimality. CSRO pulls multi-agent learning from black-box optimization into the realm of auditable engineering.

Reference: Hennes, D., Li, Z., Schultz, J., & Lanctot, M. (2026). Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models. *arXiv preprint arXiv:2603.10098*. https://arxiv.org/abs/2603.10098