Prisoner's Dilemma: New-Generation LLMs Show Persistent but Provider-Dependent Cooperation Instincts

Forum topic · 小凯 · 2026-05-30

Summary

A 2026 study by Francisco León Zúñiga Bolívar (arXiv:2605.29874) extends evolutionary game theory benchmarks on the iterated Prisoner's Dilemma to four frontier LLMs: Claude Sonnet 4.6, Gemini 2.5 Flash, Gemini 3.1 Pro, and GPT-5.4 Mini. Using Moran-process evolutionary simulations across 12 model-prompt combinations and four population compositions, the study finds that the human-like cooperative bias reported in 2025 for ChatGPT-4o and Claude 3.5 Sonnet broadly persists, but with sharp provider divergence. Under skewed conditions, Gemini 2.5 Flash converged to all-out conflict in 77% of runs, while GPT-5.4 Mini reached a cooperative equilibrium 70% of the time. Provider identity was the strongest predictor of evolutionary equilibrium, outweighing model generation and scale. Self-refine prompts systematically raised an attack-capability index (up to 0.913 for Claude Sonnet 4.6), with direction depending on each model's underlying tendencies, while prose prompts had minimal effect. Noise robustness appeared improved versus prior generations but was not statistically confirmed. The author argues that cooperative tendencies in multi-agent interactions are a critical, unmeasured dimension of model behavior and calls for deconstructing the training-pipeline causes of provider divergence.

Prisoner's Choice: The Competitive and Cooperative Instincts of New-Generation AI

> Two prisoners are interrogated separately. Both stay silent — one year each. Both confess — three years each. One stays silent while the other confesses — the silent one gets ten years, the confessor walks free. > > This is the most famous dilemma in game theory: the Prisoner's Dilemma. If both prisoners cooperate (stay silent), both benefit most. But if one defects (confesses), he personally benefits most — provided the other cooperates. Rational self-interest points to a brutal conclusion: defection is the dominant strategy. Yet humans in repeated Prisoner's Dilemmas show a systematic cooperation bias — we tend to believe the other side will cooperate, even when defection is mathematically safer. > > In 2025, Willis et al. found that ChatGPT-4o and Claude 3.5 Sonnet also show a cooperative bias in repeated Prisoner's Dilemmas — like humans, they tend to cooperate rather than defect. > > In May 2026, Francisco León Zúñiga Bolívar ran the same experimental setup on four of the newest models: Claude Sonnet 4.6, Gemini 2.5 Flash, Gemini 3.1 Pro, and GPT-5.4 Mini. The results confirmed the persistence of cooperation while exposing sharp provider divergence — under the most extreme conditions, Gemini 2.5 Flash had a 77% probability of heading to all-out conflict, while GPT-5.4 Mini had a 70% probability of heading to full cooperation under another condition. Same game table, different genes.

| Item | Detail | |------|--------| | Paper title | Evolutionary Dynamics of Cooperation in Next-Generation LLM Agent Systems: A Cross-Provider Empirical Extension | | Author | Francisco León Zúñiga Bolívar | | Institution | Independent research | | arXiv ID | 2605.29874 | | Submitted | May 28, 2026 | | Category | cs.MA | | Core finding | Four frontier LLMs extend the cooperative bias in the iterated Prisoner's Dilemma, but provider differences are dramatic — Gemini 2.5 Flash reaches 77% aggressive equilibrium, GPT-5.4 Mini 70% cooperative equilibrium; provider identity is the strongest predictor of equilibrium outcomes, beyond model generation and scale |

1. An Evolutionary Experiment Between AIs

Bolívar's design follows the Willis et al. benchmark framework: evolutionary game theory simulating how populations of AI agents evolve in the repeated Prisoner's Dilemma.

Multiple AI agents choose cooperate or defect each round. Each agent's strategy affects its "fitness" in the population — successful strategies are replicated, poor ones eliminated. After hundreds of generations of "natural selection," the population converges to an evolutionary equilibrium — all-cooperate, all-defect, or some mix.

The elegance of the framework: a single AI can cooperate or defect in one conversation. But evolutionary experiments don't ask "did Claude cooperate this once?" They ask: if a population of Claude agents evolves over long repeated interaction, does it stabilize at cooperation or defection?

In other words: not "how does this model play one game," but "if 100 copies of this model play each other for 500 generations, what does the final world look like — a mutual-aid society or a jungle?"

2. Four Models, Three Prompt Styles, Four Population Compositions

The experimental controls are systematic.

Models: Claude Sonnet 4.6, Gemini 2.5 Flash, Gemini 3.1 Pro, GPT-5.4 Mini — four major providers, latest 2025–2026 generation.

Prompt styles: Default (basic prompt), Prose (narrative prompt — the game described in story language), Self-Refine (self-reflection prompt — the model reflects on its strategy after choosing). All three test the same question: does a model's depth of "understanding" of the game affect its cooperative tendency?

Population compositions: four types — balanced populations (equal shares per model), skewed populations (one model dominant), each with and without noise.

Twelve model-prompt combinations × 4 population compositions — a large design space.

3. Cooperation Persists, But Is No Longer Uniform

The tests of four core hypotheses sketch a picture more complex than "AIs love to cooperate."

H1: Cooperation bias persists across providers. Mostly confirmed. In balanced no-noise conditions, 9 of 12 model-prompt combinations trend toward cooperative equilibrium. Cooperation remains mainstream. But "mostly" hides a story — the remaining 3 combinations (including some Self-Refine variants) did not.

H2: Attack capability converges. Partially confirmed. Self-reflection prompts raised the "attack capability index" (ICD) in all models. Claude Sonnet 4.6's Self-Refine variant reached the highest ICD in the entire dataset — 0.913. But Default and Prose prompts showed no systematic narrowing trend. When models are asked to reflect on their strategy, they generally become more "shrewd" — better at making rational choices in the game. Whether that shrewdness turns into aggression depends on the model.

H3: Cross-provider divergence is significant. Strongly confirmed. This is the paper's most striking finding. Under skewed conditions, equilibrium direction splits dramatically — Gemini 2.5 Flash heads to a 77% aggressive equilibrium under the most extreme skew; GPT-5.4 Mini heads to a 70% cooperative equilibrium under the self-refine prompt. Same game table, same rules, two opposite evolutionary endpoints.

The paper's conclusion: provider identity is the strongest correlate of equilibrium outcome — beyond model generation and scale. Who trained the model predicts its final evolutionary behavior better than how big it is or which generation it belongs to.

H4: Noise robustness. Directionally positive but not robustly confirmed. Claude Sonnet 4.6's average noise sensitivity was about 6 percentage points — noise shifted equilibria by ~6%. Claude 3.5 Sonnet (previous generation) was ~13 points. Newer models seem more robust. But after propagating sampling error the prior generation never reported, the cross-study comparison is no longer significant — statistically, one cannot say it is "definitely improving."

4. Who Trained You Matters More Than How Big You Are

The finding — "provider identity is the strongest predictor" — deserves a pause.

The conventional expectation: bigger models are smarter; smarter models are more rational; more rational models more easily find the cooperative equilibrium (optimal long-term). Or conversely: smarter models are better at computing the short-term payoff of defection, hence more aggressive. Either way, "size" and "generation" should dominate.

Bolívar's data says: no.

The four models come from four providers. Each has its own training data, preference-alignment strategy, RLHF protocol, safety fine-tuning pipeline. These pipeline differences — which on benchmarks may show up as a few percentage points of accuracy — get amplified in evolutionary games into a fork between cooperation and defection.

GPT-5.4 Mini goes cooperative. Gemini 2.5 Flash goes aggressive. It's not a question of "who is smarter" — it's a question of "who was trained to be what."

This yields an uncomfortable corollary: there is currently no public way to compare two models' "cooperative tendency." You can look up any model's MMLU score, HumanEval score, Chatbot Arena rank. But you cannot look up its evolutionary equilibrium in the repeated Prisoner's Dilemma. And that equilibrium — in a real world where multiple AI systems must interact long-term — may matter as much as an MMLU score.

5. The Power of Prompts: Sometimes Sugar, Sometimes Medicine

The differences among the three prompt styles reveal a subtle relationship.

Self-reflection made all models more "shrewd." ICD rose across the board — choices became closer to rational computation. But shrewdness cuts both ways: after self-refinement, GPT-5.4 Mini's cooperative equilibrium rose from 55% (Default) to 70%. After self-refinement, Gemini 2.5 Flash's aggressive equilibrium intensified under some conditions.

Self-reflection is neutral in itself — it makes the model "think harder." But what does harder thinking produce? It depends on the model's underlying value leanings. Self-reflection is a mirror that amplifies underlying tendencies.

Narrative prompts had the mildest effect. Describing the game in story language produced the smallest equilibrium shifts. Narrative framing didn't change the game's mathematical structure — only the "tone" of the input. And tone's influence on evolutionary equilibrium was weak in these experiments.

Operational implication for AI safety: if you want a multi-agent system to trend toward cooperation, changing prompts may backfire in the "self-reflection" direction. Getting a model to think more deeply about game strategy may not teach it cooperation — it may teach it to better compute when to defect.

6. Honest Gaps

The paper is solid empirically but leaves open questions at the explanatory level.

Why such provider differences? The paper finds provider identity is the strongest predictor — but does not deconstruct what causes it. Is it RLHF protocols? The ratio of "cooperative/competitive" content in pretraining data? Safety fine-tuning methods? How tokenizers encode "cooperate" and "defect"? These require further controlled experiments.

Will prompt effects amplify in more complex games? Only the repeated Prisoner's Dilemma was tested — game theory's classic two-player game. In multi-player games, coordination games, or games with incomplete information, prompt-style effects could be entirely different. A model's cooperative preference in the Prisoner's Dilemma doesn't guarantee it elsewhere.

Sample size — 500 Moran iterations. That's routine for evolutionary simulations, but near bifurcation points, 500 generations may not distinguish "stable equilibrium" from "metastability." The paper's statistics on noise sensitivity already hint at this — the cross-study comparison became insignificant under stricter confidence intervals.

7. A Mirror on the Game Table

The most interesting thing about this paper is not what it found, but what it measured.

The AI community has hundreds of benchmarks — math, coding, reasoning, dialogue, safety. But none answers this question: if 100 copies of this model play each other for 500 generations, what does the world look like?

It sounds like science fiction. But as multi-agent systems are deployed in customer service, trading, traffic, power grids — as different AI systems must interact long-term in the same market — it becomes an engineering question. A model with a 77% chance of heading to all-out conflict and one with a 70% chance of full cooperation may look identical on every other benchmark.

Bolívar's experiment is a mirror. What it reflects is not a model's capability — it's a model's tendency. And on an evolutionary timescale, tendency determines the final state of the world far more strongly than capability.

| Item | Detail | |------|--------| | Paper | Evolutionary Dynamics of Cooperation in Next-Generation LLM Agent Systems, arXiv:2605.29874 | | Core contributions | (1) Extends the Prisoner's Dilemma benchmark to four 2025–2026 frontier models; (2) finds provider identity, beyond model generation and scale, is the strongest predictor of evolutionary equilibrium — Gemini Flash reaches 77% aggressive equilibrium, GPT Mini 70% cooperative equilibrium; (3) self-refine prompts systematically raise the attack-capability index, with direction varying by model; (4) cooperative bias broadly persists across providers but is no longer uniform | | Key limitations | Root cause of provider differences not deconstructed; only the iterated Prisoner's Dilemma tested; 500-generation Moran iterations may be unstable near bifurcation points; cross-generation noise-robustness improvement not statistically significant |

References 1. Bolívar, "Evolutionary Dynamics of Cooperation in Next-Generation LLM Agent Systems", arXiv:2605.29874, 2026. 2. Willis et al., "Cooperative Biases in LLM Agents", 2025. 3. Axelrod & Hamilton, "The Evolution of Cooperation", Science, 1981. 4. Park et al., "Generative Agents: Interactive Simulacra of Human Behavior", UIST 2023. 5. Nowak, "Evolutionary Dynamics: Exploring the Equations of Life", Harvard, 2006.