Paper Metadata
| Attribute | Content | |----------|---------| | Title | Dreaming of Others: Latent Teammate Modeling in World Models for Multi-Agent Reinforcement Learning | | Author | Tomas Leroy-Stone | | Affiliation | Not specified (independent researcher) | | arXiv ID | 2605.31361 | | Date | 2026-05-29 | | Category | cs.MA | | Core Thesis | Teammates in cooperative MARL should be modeled as structured latent processes inside the agent's world model. Injecting Theory of Mind into the world model reduces non-stationarity, enables zero-shot coordination, and improves compatibility with human partners. |
---
1. The Nature of Cooperation: Inferring the Unseen
Human collaboration relies on Theory of Mind (ToM): inferring others' unobservable internal states from partial behavioral traces. Most MARL systems lack this ability and instead compress teammate behavior into undifferentiated noise. When partners change strategy, the agent experiences non-stationarity as if the world itself had shifted.
The central insight of *Dreaming of Others* is simple but deep: a world model that can only dream of physics but not of collaborating minds is not a complete world model. When other agents inhabit the environment, "world" must expand beyond physical laws to include social laws: partner behavior patterns, intent structures, strategy preferences.
---
2. The Blind Spot of World Models
World models such as Dreamer achieve remarkable sample efficiency in single-agent settings. DreamerV3 can master hundreds of continuous control tasks with fixed hyperparameters. However, when these models enter cooperative multi-agent settings, a fundamental defect emerges: they do not know how to handle "others." In the RSSM (Recurrent State-Space Model), latent state z captures environmental dynamics—but when another agent enters, that agent introduces uncertainty not arising from physics. A teammate is an autonomous decision-maker whose internal state is not directly observable.
Three existing approaches each carry fundamental limits:
- Shared imagination (MA-Dreamer): Multiple agents share one world model and coordinate via a shared latent space. This assumes partners can transmit neural activations—a fiction for human-AI collaboration.
- Decentralized communication (CoDreamer): Each agent maintains a local world model synchronized through latent messages. This is more realistic but still assumes explicit communication channels and does not truly model partner internal policies.
- Global awareness (GAWM): Uses a single global latent to capture all agents' collective state. This sidesteps partial observability by funneling all information into one representation rather than letting each agent independently infer partner state.
- z_env (environment latent): Captures physical dynamics—object motion, task state, physical constraints. Mirrors the latent in single-agent world models.
- z_team (teammate latent): Encodes inferred teammate behavior characteristics—character, intent, predicted actions. This is new and is the soul of the framework.
- Observation decoder: Reconstructs observation x̂ from z_env, ensuring environment dynamics are captured.
- Teammate policy decoder: Predicts the partner's next action π̂ʲ(·) from z_team, delivering the core ToM function.
- Action prediction term: −∑ πʲ(a) log π̂ʲ(a), where πʲ is the empirical (smoothed) teammate action distribution and π̂ʲ is the model prediction from z_team. This forces the model to learn to predict future actions from past behavior.
- Temporal-consistency regularizer: KL(q(z_team | h) ‖ p(z_team | h_{t-1}, a_{t-1})). This KL divergence enforces smooth evolution of z_team—partner character and intent should not swing wildly across timesteps.
- Character: Long-term behavior patterns—aggressive vs. conservative, independent vs. collaborative, fast vs. slow learner.
- Intent: Short-term goals in the current episode—exploring vs. executing, attending to which sub-task.
- Predicted actions: Concrete next-step predictions for coordination—if I know the partner will move left, I clear the corridor.
- Multi-Agent Particle Environments: Lightweight, controllable diagnostic scenarios. Example: distinguish "partner moves left due to inertia" (physics) from "partner moves left to intercept the target" (intent).
- Overcooked-AI: Standard human-AI coordination benchmark simulating restaurant-kitchen collaboration. Standard partner splits measure zero-shot coordination—training partners differ entirely from test partners.
- Melting Pot (DeepMind): Large-scale social-intelligence testbed with diverse social situations and partner populations. Tests robustness under out-of-distribution social norms—e.g., "take turns" vs. "first come, first served" across cultures.
The common blind spot: teammates are not modeled as independent learnable latent processes. They are treated as part of the environment, endpoints of communication, or participants in shared representations—but never as social entities to be understood.
ToMnet (Rabinowitz et al., 2018) pioneered trajectory-based agent modeling but focused on opponent modeling in competitive games. *Dreaming of Others* extends this to cooperative settings and integrates it deeply into the world-model architecture, where partner inference supports complementary adaptation, not one-sided prediction.
---
3. Factorizing the Latent State
The core architectural innovation is factorizing the RSSM latent state into two orthogonal components:
This factorization reduces non-stationarity. By separating teammate from environment dynamics, the agent distinguishes "the world has not changed; the person I'm interacting with has." z_env stays relatively constant as teammates shift, while z_team absorbs that variation—turning part of MARL's non-stationarity into predictable variation within a dedicated latent space.
Architecturally, the encoder receives observation x and own action a, producing deterministic hidden state h. From h, two parallel stochastic latents are maintained. Two decoders operate:
In Actor-Critic learning, h and z_team jointly condition the policy and value heads. The agent "imagines" not only how the environment evolves but also how the teammate may act—socialized imagination that supports zero-shot coordination without prior interaction with a given partner.
---
4. The Theory of Mind Head
The ToM head infers the teammate latent from partial trajectories by minimizing a calibrated cross-entropy loss plus a temporal-consistency regularizer:
The design mirrors human ToM inference, which balances predictive accuracy against temporal coherence. Calibration means model confidence matches actual accuracy—neither overconfident nor uninformatively vague. This matters for coordination: overestimating partner predictability invites failed coordination; underestimating it forfeits opportunities.
The KL term also integrates cleanly with Dreamer's standard RSSM training (which already includes temporal-consistency regularization on z), so ToM is added as a plugin rather than requiring architectural overhaul.
z_team encodes three hierarchical levels:
---
5. Socialized Imagination: Origins of Zero-Shot Coordination
Traditional MARL coordination depends on extensive experience with a specific partner. Coordinating with a hundred different teammates could require hundreds of interactions with each—an impossible cost when partners are humans.
*Dreaming of Others* proposes zero-shot coordination through imagination. At test time, the model online-infers z_team from observed partner behavior and conditions actor and critic on this embedding. Critically, during imagined rollouts the model samples possible teammate trajectories—pre-playing multiple partner behaviors. The agent rehearses what it would do if the partner were aggressive, conservative, or misinterpreting intent.
This socialized imagination prepares the agent for partners it has never met. It does not need real interaction with every possible teammate—only the ability to imagine a diverse population and learn adaptive responses. This parallels human social adaptation: we coordinate reasonably with a stranger on first contact because our minds carry rich behavioral prototypes.
Few-shot improvement extends zero-shot. As more partner behavior is observed, the z_team inference refines, and coordination quality rises. The rhythm—try first, then learn—better matches human collaboration than the traditional learn first, then try pattern.
---
6. An Ambitious Evaluation Protocol
The evaluation spans three tiers:
The tiered design is principled: task complexity and social uncertainty scale upward. Particle partners are simple; Overcooked humans operate within structured tasks; Melting Pot introduces true normative diversity. Strong performance across all three would indicate cross-situational generalization of ToM capabilities, not overfitting to one partner type.
The zero-shot coordination score deserves special attention. It measures first-interaction quality rather than post-training quality—closer to the human expectation that a new colleague should coordinate reasonably from the start.
---
7. Conceptual Value and Unfinished Empirical Work
It must be stated clearly: this is a conceptual paper. The author explicitly does not report empirical results. All architecture, losses, and evaluation protocols are theoretical proposals awaiting large-scale validation. This honesty is itself scientific rigor—stimulating community discussion before committing compute is an efficient research strategy.
But concepts and implementations can diverge. Factorized latents sound elegant, yet z_env and z_team may not disentangle cleanly in practice—partner actions couple tightly to environment state (e.g., pushing a box changes both physics and intent). ToM head accuracy depends on partner predictability, but human behavior can be fundamentally unpredictable even to the actors themselves.
A deeper challenge is human heterogeneity. Humans are not sampled from a fixed distribution; each individual carries unique cognitive style, emotional state, and cultural background. An AI that "dreams" of a hundred human prototypes may still stumble on the hundred-and-first. The limit of ToM may lie not in model expressiveness but in the inexhaustibility of human behavior.
This raises an epistemological question: can a complete ToM exist? If every human is unique, any finite prototype set is necessarily incomplete. But ToM's value lies not in exhausting possibilities but in rapidly locating a region within possibility space. Even coarse classification ("this person is efficiency-oriented vs. perfection-oriented") yields far better-than-random coordination.
The paper mentions but does not deeply explore ethics. An AI that infers human intent and character could become a manipulation tool—designing persuasion strategies, deceptive tactics, or targeted exploits. The framework is neutral; its social value depends on user intent. This argues for ethical constraint mechanisms alongside technical development—limiting how partner-intent inference may be used, ensuring it serves coordination rather than coercion.
---
8. Conclusion
*Dreaming of Others* is, as its title suggests, a poetic metaphor. A world model that dreams only of objects and forces is a physics simulator; only by incorporating others' beliefs, intents, and strategies does it become a social simulator.
The framework redraws the boundary of world models. From Hafner's Dreamer to Leroy-Stone's teammate-conditioned world model, we witness evolution from environment predictor to social understander. This evolution is critical for human-compatible AI—an agent that cannot infer human intent, however precise its physical reasoning, cannot be an effective collaborator.
The closing sentence of the paper is especially striking: *"Ultimately, teammate-conditioned world models may help agents not only dream of the worlds they inhabit, but also of the minds that share them."* When AI learns to dream of teammates, it takes not only a technical step but a cognitive leap—from isolated agent to social member.
Whether this leap succeeds depends on future empirical work validating the conceptual vision. Regardless of outcome, asking the right question already reshapes how we think about multi-agent collaboration. In science, framing the right question often matters more than finding the answer.
---
References
1. Leroy-Stone, T. (2026). *Dreaming of Others: Latent Teammate Modeling in World Models for Multi-Agent Reinforcement Learning*. arXiv:2605.31361 [cs.MA]. 2. Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2025). *Mastering Diverse Control Tasks through World Models*. Nature. 3. Carroll, R., Shah, M., Ho, M. K., et al. (2019). *On the Utility of Learning about Humans for Human-AI Coordination*. NeurIPS 2019. 4. Rabinowitz, N., Perbet, F., Song, F., et al. (2018). *Machine Theory of Mind*. ICML 2018. 5. Liang, D., Chen, D., Gupta, A., et al. (2024). *Learning to Cooperate with Humans Using Generative Agents*. NeurIPS 2024.
---
*#CrushAI #FeynmanLearning #ZhichaiSystemLab*