Overview
As LLMs become more widely deployed, they are increasingly expected to work alongside other AI agents rather than operating in isolation. Effective coordination in these settings requires agents to communicate, share information, and make decisions under uncertainty.
SMAC-Talk is a natural language extension of the StarCraft Multi-Agent Challenge, introduced by Joel Sol and Homayoun Najjaran for evaluating LLM-based agents in cooperative multi-agent environments.
- Paper (arXiv): 2506.00634
- Authors: Joel Sol, Homayoun Najjaran
- Published: 2025-06-01
- Field: Machine Learning
- Decentralized control: agents operate independently without centralized oversight
- Partial observability: agents have incomplete information about the environment
- Long-horizon decision making: tasks require sustained strategic planning
- Natural language communication channel: used to probe coordination and trust between agents
- reasoning structure,
- memory, and
- model scale
Key Features
Evaluation Scenarios
The communication channel enables varied evaluation setups, including scenarios with an embedded deceptive agent — one that communicates only to disrupt and mislead its allies. This allows researchers to measure robustness against manipulation and the quality of trust between LLM agents.
Benchmark Setup
The authors provide three agents for benchmarking, using four models from the Qwen family, and study how:
SMAC-Talk is released as an open benchmark to support the research community in developing and evaluating LLM agents in cooperative multi-agent scenarios.