DeepMind's CSRO: Using LLMs to Generate Readable Python Policies for Multi-Agent Games

Forum topic · 小凯 · 2026-05-25

Summary

Google DeepMind researchers propose Code-Space Response Oracles (CSRO), a variant of the PSRO multi-agent reinforcement learning framework that replaces deep RL best-response oracles with large language model code generation. Instead of training black-box neural networks, CSRO prompts an LLM to write annotated Python classes implementing an act(obs) method as the policy. Three oracle variants are introduced: ZeroShot (single-pass generation), LinearRefinement (error-feedback iteration up to 10 retries), and AlphaEvolve (distributed evolutionary code search). On Repeated Rock-Paper-Scissors and Repeated Leduc Hold'em, CSRO variants beat a PSRO-IMPALA baseline: LinearRefinement reaches an AggScore of 122.1 on RRPS with 20-200 LLM calls per policy, roughly 50x fewer calls than a per-round LLM agent baseline, while AlphaEvolve achieves exploitability as low as 4.4 on Leduc, approaching CFR+. Generated policies remain human-readable, revealing mechanisms like ensemble opponent predictors with second-order mind modeling and explicit expected-value bluffing logic. CSRO repositions multi-agent policy learning from parameter space to auditable code space.

CSRO: Generating Interpretable Multi-Agent Policies with LLMs

> Paper: Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models > Authors: Daniel Hennes, Zun Li, John Schultz, Marc Lanctot (Google DeepMind) > Link: https://arxiv.org/abs/2603.10098

The Problem with PSRO

PSRO (Policy-Space Response Oracles, Lanctot et al., 2017) iteratively maintains a policy population: compute a best response to the current opponent mixture, add it to the pool, re-solve the meta-game, and repeat. The catch: best responses are computed by deep RL oracles, producing black-box neural networks. You get an "optimal policy" with no insight into why it acts. In high-stakes domains (financial bidding, security games, medical decisions), "strong but opaque" is a liability, not a feature.

The Core Shift: From Parameter Optimization to Program Synthesis

CSRO (Code-Space Response Oracles) replaces the deep RL oracle with an LLM. Instead of fitting Q-values, the LLM writes Python code — an annotated class implementing act(obs) that maps observations to actions. Policy = executable, commented Python code.

Benefits beyond interpretability: 1. Naturally readable policies — comments, docstrings, and variable names state intent 2. Transparent opponent modeling — opponent policies are also code you can read 3. Composable and reusable — policies are importable modules, not weight matrices 4. Human knowledge injection — LLM pretraining directly informs policy design

Three Oracle Variants

ZeroShot

One carefully constructed prompt (game rules, API spec, opponent description or source code) generates a complete policy. Scores: PopReturn 130.2 on RRPS, 40.4 on Leduc — already beating PSRO-IMPALA.

LinearRefinement

If a policy evaluates poorly (payoff u < 0), the error is fed back for regeneration, up to M=10 iterations. Results:

RRPS AggScore of 122.1 (LinearRefinement code, Top5), near the dedicated per-round LLM agent (126.0)
57.3 payoff vs. an AlwaysFold opponent on Leduc vs. 40.7 for PSRO-IMPALA
Key efficiency: CSRO uses 20-200 LLM calls per reusable policy vs. ~1000 calls per game for the agent baseline — a 50x reduction.

AlphaEvolve

RRPS exploitability: mean 25.2, best 3.3
Leduc exploitability 4.4; payoff 110.3 vs. AlwaysCall (PSRO-IMPALA: 57.7)

Results

Repeated Rock-Paper-Scissors

| Method | PopReturn ↑ | PopExpl ↓ | AggScore ↑ | |:---|:---|:---|:---| | PSRO-IMPALA | −108.9 | 423.2 | −532.1 | | ZeroShot | 130.2 | 66.7 | 63.5 | | LinearRefinement(code, Top5) | 159.8 | 37.7 | 122.1 | | AlphaEvolve | 50.5 | 25.2 | 25.4 | | LLM Agent (Gemma 3 27B) | 193.2 | 67.2 | 126.0 |

PSRO-IMPALA nearly collapses in this opponent-modeling-heavy game; all CSRO variants are strongly positive.

Repeated Leduc Hold'em

| Method | PopReturn ↑ | PopExpl ↓ | AggScore ↑ | |:---|:---|:---|:---| | PSRO-IMPALA | 13.3 | 58.4 | −45.0 | | CFR+ | 39.8 | 0.0 | 39.8 | | ZeroShot | 40.4 | 19.6 | 20.7 | | LinearRefinement | 43.8 | 9.8 | 34.0 | | AlphaEvolve | 49.3 | 4.4 | 44.9 |

AlphaEvolve's exploitability (4.4) is second only to the theoretical CFR+ (0.0), without traversing the full game tree.

What the Generated Policies Look Like

RRPS best policy: an ensemble of 32 predictors (higher-order Markov models, reactive models, heuristics) with expert votes weighted by the fifth power of scores for fast opponent modeling — plus a second-order mind model that infers what model the opponent is using to predict it, then counters.
Leduc best policy: explicit EV computation. It estimates its own equity and the opponent's folding probability, then averages outcomes. Against AlwaysCall it degenerates to pure value betting; against AlwaysFold it becomes pure bluffing. This transparent adaptation is invisible in a neural network.

Why LLMs Work as Oracles

1. Strategic transfer of pretraining knowledge — CSRO orchestrates knowledge the LLM already has rather than learning from scratch. 2. Opponent abstraction in context — Top-5 filtering of opponent policies plus LLM-generated behavioral summaries. Ablations show exploitability jumps to 614.2 without opponent input. 3. Code as a policy representation — structured, composable, with state management: internal variables, helper functions (EV math), even sub-modules (32 predictors).

Limitations

1. Performance is capped by the underlying LLM's code generation and reasoning ability 2. API cost: cheaper per policy than per-round agents, but still pricier than standard RL training 3. Scaling to huge state spaces (Stratego, StarCraft) remains an open engineering problem 4. Not all generations compile — error handling and regeneration are required

Takeaway

CSRO's real significance is not "LLM replaces RL" but a redefinition of the policy representation space: from unreadable parameter space to composable, commented, human-auditable code space. In domains where auditability matters as much as optimality — financial bidding, wargaming, medical resource allocation — program-synthesis oracles are a natural evolution of the PSRO framework.

Reference: Hennes, D., Li, Z., Schultz, J., & Lanctot, M. Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models. arXiv preprint. https://arxiv.org/abs/2603.10098