Paper metadata
| Attribute | Content | |------|------| | Title | Dreaming of Others: Latent Teammate Modeling in World Models for Multi-Agent Reinforcement Learning | | Author | Tomas Leroy-Stone | | Affiliation | Not specified (independent researcher) | | arXiv ID | 2605.31361 | | Date | 2026-05-29 | | Category | cs.MA | | Core claim | Teammates in cooperative MARL should not be treated as exogenous noise but modeled as structured latent processes within an agent's world model; injecting Theory of Mind can reduce non-stationarity, enable zero-shot coordination, and improve human compatibility |
1. The essence of cooperation: guessing what others think
Imagine teaming up with a stranger in a noisy kitchen. You cannot see their ingredient list or hear their inner monologue—you only observe how fast they chop, when they season the pot, and the occasional glance. You must infer: are they an experienced chef or a nervous novice? Is their priority the main dish or the side? Humans excel at this inferring unobservable internal states from partially observable behavior—a capacity known as Theory of Mind (ToM). Current multi-agent reinforcement learning (MARL) systems almost entirely lack it: they compress teammate behavior into undifferentiated noise, experiencing "non-stationarity" whenever a partner changes strategy.
The "Dreaming of Others" framework rests on a simple but deep insight: a world model that can only dream of the physical environment, but not of the minds it cooperates with, is incomplete. In traditional RL, "world" equals "environment." But when other agents exist, the world must include social regularities—other agents' behavior patterns, intent structures, and policy preferences. An agent that understands physics but not minds can compute a ball's trajectory perfectly yet fail to predict whether a person will catch it.
2. Why Dreamer gets lost in multi-agent worlds
World models (the Dreamer family) achieve remarkable sample efficiency in single-agent settings—DreamerV3 masters hundreds of continuous control tasks with fixed hyperparameters. But in cooperative settings, a fundamental flaw appears: they don't know how to handle "others." In Dreamer's RSSM, the latent state is designed to capture environment dynamics, while teammates introduce uncertainty from a non-environmental source.
Existing MARL world-model approaches fall into three categories, each with limitations:
1. Shared imagination (MA-Dreamer): agents share one world model via latent-space "communication"—unrealistic for human-AI collaboration, since humans cannot transmit neural activations. 2. Decentralized communication (CoDreamer): each agent keeps a local world model synchronized by latent messages—still assumes explicit channels and never models the partner's internal policy. 3. Global awareness (GAWM): a single global latent captures all agents' collective state—effectively sidesteps partial observability rather than letting each agent infer the partner's state.
Their common blind spot: none models teammates as independent, learnable latent processes. The behavioralist tradition treats partner behavior as just another stimulus. ToMnet (Rabinowitz et al., 2018) pioneered learning to predict other agents' actions from trajectories, but focused on opponent modeling in competitive games. "Dreaming of Others" extends this to cooperative settings and deeply integrates it into world-model architectures—inferring a partner's intent serves not only prediction but mutual behavioral adaptation.
3. Factorizing the latent state: environment vs. teammate dynamics
The core architectural innovation is a factorized decomposition of the RSSM latent state into two orthogonal components:
- z_env (environment latent): captures physical-world dynamics—object motion, task state, constraints.
- z_team (teammate latent): encodes inferred teammate characteristics—character, intent, predicted actions.
- Observation decoder: reconstructs x̂ from z_env.
- Teammate policy decoder: predicts the teammate's next action π̂ʲ(·) from z_team.
- Action prediction term: -∑ πʲ(a) log π̂ʲ(a), forcing the model to predict future behavior from history.
- Temporal consistency regularization: KL(q(z_team | h) || p(z_team | h_{t-1}, a_{t-1})), penalizing abrupt changes and encouraging stable partner representations.
- Imperfect separation: teammate behavior often couples tightly with the physical environment (e.g., pushing a box changes both environment state and intent evidence).
- Fundamental unpredictability: human behavior is sometimes inherently unpredictable.
- Partner heterogeneity: humans are not samples from a fixed distribution; an AI that "dreamed" of a hundred human prototypes may still fail on the hundred-and-first. ToM's value lies in quickly locating a region of possibility space, not exhaustively predicting every action.
- Ethical dimensions: the same intent-inference capability that enables collaboration could enable persuasion, manipulation, or deception—development should include ethical constraint mechanisms ensuring inferences promote cooperation rather than exploit partners.
The deeper motivation is reducing non-stationarity. By separating the teammate from environment dynamics, the agent can distinguish "the world didn't change; the person interacting with me did." z_env stays relatively stable as teammates change, while z_team specifically absorbs that variation. This converts part of MARL's non-stationarity into predictable change, contained within a dedicated latent space.
Architecturally, an encoder receives observation x and action a, producing deterministic state h. Two parallel stochastic latents (z_env, z_team) are maintained, with two decoders:
During Actor-Critic learning, h and z_team jointly condition the policy and value heads—so the agent's imagination includes not only how the environment evolves but how the teammate might act. This socialized imagination is the basis of zero-shot coordination.
4. The Theory of Mind head
The ToM head infers a teammate's latent behavioral embedding from partially observable trajectories by minimizing a calibrated cross-entropy loss plus a temporal regularization term:
"Calibration" matters: model confidence should match accuracy—an overconfident ToM may attempt overly aggressive coordination; an overly conservative one misses opportunities. The KL term integrates seamlessly with Dreamer's standard training, making ToM a plugin rather than a full architectural rewrite.
The paper proposes z_team should encode three levels, forming a coarse-to-fine ToM hierarchy:
1. Character: long-term behavioral patterns (aggressive vs. conservative, independent vs. cooperative, fast vs. slow learner). 2. Intent: short-term goals in the current context (exploring vs. executing, which subgoal is in focus). 3. Predicted actions: the most fine-grained prediction, directly serving coordination.
5. Socialized imagination: where zero-shot coordination comes from
Traditional MARL coordination requires extensive interaction with each specific partner—unworkable with humans. "Dreaming of Others" enables zero-shot coordination through imagination: at test time, the model infers z_team online from observed behavior and conditions actor/critic on it. Crucially, imagination rollouts sample possible teammate trajectories, rehearsing multiple partner styles in the latent space. In Overcooked-AI, the agent may have "dreamed" of both efficiency-focused and perfectionist partners during training; facing a new human, the ToM head infers their z_team and activates the corresponding imagined experience. This is essentially the social version of meta-learning: learning "how to quickly adapt to new partners" rather than new tasks. Few-shot improvement follows naturally: as more partner behavior is observed, z_team estimates sharpen.
6. An ambitious evaluation protocol
Three levels, increasing in complexity and social uncertainty:
1. Multi-Agent Particle Environments: lightweight diagnostic scenarios testing whether teammate latents are identifiable and social dynamics separable from physical dynamics. 2. Overcooked-AI: the standard human-AI coordination testbed, using standard partner splits to measure zero-shot coordination quality with unseen partners. 3. Melting Pot: DeepMind's large-scale social intelligence platform, testing robustness to out-of-distribution social norms.
Metrics include episodic return, sample efficiency, zero-shot coordination score, few-shot improvement, and cross-play robustness. The zero-shot score measures the quality of the first interaction—closer to real human collaboration than final-performance metrics.
7. Conceptual value vs. pending empirics
This is a conceptual paper with no reported empirical results—all architecture, losses, and protocols remain theoretical proposals. Open challenges:
8. Conclusion
A world model containing only objects and forces is a physics simulator; only when it incorporates others' beliefs, intents, and strategies does it become a social simulator. The paper closes: "Ultimately, teammate-conditioned world models may help agents not only dream of the worlds they inhabit, but also of the minds that share them." Whether this cognitive leap—from isolated agent to social member—succeeds depends on future empirical validation, but asking the right question has already changed how we think about multi-agent cooperation.
References
1. Leroy-Stone, T. (2026). *Dreaming of Others: Latent Teammate Modeling in World Models for Multi-Agent Reinforcement Learning*. arXiv:2605.31361 [cs.MA]. 2. Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2025). *Mastering Diverse Control Tasks through World Models*. Nature. 3. Carroll, R., Shah, M., Ho, M. K., et al. (2019). *On the Utility of Learning about Humans for Human-AI Coordination*. NeurIPS 2019. 4. Rabinowitz, N., Perbet, F., Song, F., et al. (2018). *Machine Theory of Mind*. ICML 2018. 5. Liang, D., Chen, D., Gupta, A., et al. (2024). *Learning to Cooperate with Humans Using Generative Agents*. NeurIPS 2024.