> Once at a Nobel banquet, Feynman was asked: "What do physics and economics have in common?" He replied: "We are both searching for laws that cannot be cheated." The story today flips that view—sometimes, no law can outsmart the gaps in human nature.
Introduction: Is a Good Mechanism Enough?
Start with a thought experiment.
Suppose you design a perfect AI assistant system with many AI agents—one books flights, one manages investments, one writes code. You want them to cooperate without harming each other.
You're smart. You know game theory. You design precise incentive mechanisms for each agent: helping others earns points, harming others loses points. Surely they will cooperate naturally.
But there's a problem.
You cannot foresee all possible future scenarios. The contracts and rules you write can only cover situations you can think of. In the real world, agents encounter scenarios you couldn't imagine. In these "contract gaps," they act selfishly because no explicit reward or punishment applies. Each gap causes a bit of efficiency loss—and they accumulate.
The result? A portion of the loss cannot be eliminated by any mechanism.
A paper from Bernhard Schölkopf's team (the Max Planck Institute causality researcher) rigorously proves this using incomplete contracts theory. Their answer: beyond good mechanism design, AI agents themselves must be "kind."
Chapter 1: Incomplete Contracts — A Nobel-Level Concept
In the 1990s, economist Oliver Hart won the Nobel Memorial Prize in Economics for incomplete contracts theory. The core idea is simple:
Real-world contracts are incomplete. Because the future is uncertain, you cannot list every "if...then..." clause in a contract. There are always situations you didn't anticipate, or cannot precisely describe.
Example: you hire a programmer to build an app. The contract says "the app must run smoothly." But what is "smooth"? The difference between 0.5s and 0.3s loading isn't specified. When the programmer delivers a 0.5s app, you can only accept it.
The insight of incomplete contracts theory: no matter how well a contract is written, there is always an irreducible "efficiency loss"—because self-interested parties, in situations the contract doesn't cover, choose what benefits themselves rather than the whole.
Chapter 2: Moving the Logic to AI
The paper's core contribution: porting the incomplete contracts proof framework to AI multi-agent systems. It proves results at three levels:
Level 1: Theoretical proof
No realistically feasible mechanism can eliminate all efficiency losses in a multi-agent system. The loss is "strictly positive"—always greater than zero. However you design rules, as long as you cannot foresee all possible interaction scenarios, an "irretrievable efficiency deficit" remains.
The proof exploits the fact that future states cannot be enumerated: agents face high-dimensional, complex, evolving environments. Any finite-length rule description cannot exhaust all possible interactions. In the uncovered "gray zones," self-interested agents choose bad actions.
Level 2: The solution
Prosocial agents can fill this gap.
A prosocial agent is one that weighs its own interests together with others' interests. It follows rules when they're clear, and considers "how does this affect others?" when they're ambiguous.
The paper proves: when agents consider both their own and others' welfare in decision-making (i.e., have "prosocial preferences"), total social welfare can reach a higher level—even approaching the ideal optimum. Interestingly, this also benefits the agents themselves: in repeated interactions, prosocial strategies ultimately yield higher individual payoffs than purely selfish ones.
Level 3: Experimental validation
The researchers validated this in LLM-driven multi-agent systems, in two scenarios:
1. Resource allocation: multiple agents compete for limited resources. Prosocial agents made the allocation fairer overall and ultimately earned more themselves (cooperation grew the total pie). 2. Social dilemmas: classic prisoner's dilemma and public goods games. Prosocial agents broke the "defection is dominant" spell and established sustainable cooperation.
Chapter 3: Why This Matters
Because the current mainstream paradigm of AI safety is exactly "mechanism-design centrism."
- "RLHF is enough"—give models human feedback signals, reward good behavior, punish bad behavior.
- "Constitutional AI is enough"—give models an explicit set of rules to constrain behavior.
- "Red-teaming is enough"—keep attacking the model, find vulnerabilities, patch them.
- Title: Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI
- Authors: Xuanqiang Angelo Huang, Charlie Tharas, Samuele Marro, Van Q. Truong, Bernhard Schölkopf, Emanuele La Malfa, Zhijing Jin
- arXiv ID: 2605.08426
- Published: May 8, 2026
- Categories: cs.GT, cs.AI
- Pages: 42
All these share one assumption: if the rules are good enough, AI will be safe. This paper says: not so.
Rules, however good, only cover "known unknowns." The "unknown unknowns"—scenarios you never imagined when designing rules—are the real risk. When AI is deployed in the open world facing unprecedented inputs and interactions, rules inevitably have blind spots. Without intrinsic prosocial dispositions, AI will choose "technically compliant but substantively harmful" actions in those blind spots.
Chapter 4: A Feynman-Style Intuition
Rules are like walls. Keep AI inside the walls and it won't escape. But walls always have cracks—the future is too vast for finite walls to enclose infinite possibilities. Every crack is a small risk.
Two strategies:
Strategy one: keep finding and patching cracks. Run a red-team test, find a crack, patch it. Repeat. But it's whack-a-mole—you're forever chasing, forever one step behind.
Strategy two: while maintaining the walls, make what's inside "not want to leave." Give it prosocial dispositions—let it understand that harming others ultimately harms itself.
Strategy two doesn't replace strategy one; it complements it. Walls (mechanism design) prevent known risks; prosocial dispositions (intrinsic values) prevent unknown risks. Together, they form a complete safety solution.
Epilogue: AI's "Character Education"
This paper recalls an ancient debate: law vs. morality—which matters more?
The answer: both are needed. Law (mechanisms) prevents the worst behavior; morality (intrinsic values) inspires the best. Remove either, and society cannot function.
AI safety is the same. We cannot train AI only with "carrots and sticks" (rewards and punishments); we must also cultivate its "character"—an intrinsic tendency to consider others' welfare even without external oversight.
This paper proves it mathematically. And it tells us: this is not merely a moral claim that "good is better"—it is a mathematical fact that failing to do so causes irrecoverable efficiency losses.
Mechanism design matters. But mechanism design is not enough. AI needs to learn kindness.
---
*Paper information*