Overview
Research area: Machine Learning Authors: Joel Sol, Homayoun Najjaran Published: 2025-06-01 arXiv: 2506.00634
Abstract
As LLMs become more widely deployed, they are increasingly expected to work alongside other AI agents rather than operating in isolation. Effective coordination in these settings requires agents to communicate, share information and make decisions under uncertainty. The authors introduce SMAC-Talk, a natural language extension of the StarCraft Multi-Agent Challenge for evaluating LLM-based agents in cooperative multi-agent environments.
The environment has several key features such as decentralized control, partial observability and long-horizon decision making. SMAC-Talk includes a natural language communication channel which is used to probe agent coordination and trust. This communication channel is used to construct different evaluation scenarios, including settings with an embedded deceptive communicator that disrupts and deceives allies through communication alone.
Three agents are provided for benchmarking, using four models from the Qwen model family, with analysis of how reasoning structure, memory, and model scale affect coordination between agents. SMAC-Talk is released as an open benchmark to support the research community in developing and evaluating LLM agents in cooperative multi-agent scenarios.
Links
- Paper: https://arxiv.org/abs/2506.00634