CollabSim: Why LLM Agent Teams Fail at Collaboration, Not Capability

You've probably worked on an all-star team where every individual was top-tier, yet nothing got done. Meetings drifted, information stayed siloed, people worked in parallel. The problem wasn't individual capability—it was collaboration.

Researchers at Northeastern University and Microsoft found that LLM multi-agent systems (MAS) face exactly the same problem. Their CollabSim framework is the first to systematically diagnose the *collaborative competence* of AI agents—not whether they can complete tasks alone, but whether they can work together effectively as a team.

The Problem: Evaluations Look at Outcomes, Not Process

Current multi-agent evaluation has a blind spot: it only measures task completion. If agents team up to write code and the code runs, they get a high score. If they produce a correct research conclusion, high score.

But this misses a key question: task completion doesn't mean the collaboration was effective. Maybe one agent did 90% of the work while others coasted. Maybe information transfer was so inefficient that the same work was done three times. Maybe one agent held critical information but never told its teammates.

The CSCW (Computer-Supported Cooperative Work) field has studied human remote collaboration for over 40 years. It found that team failures are rarely due to insufficient individual skill—more often they stem from: failure to establish common ground, failure to maintain shared task understanding, failure to balance individual and collective interests, and failure to repair deviations during collaboration.

CollabSim brings these insights into AI agent evaluation.

Framework: Four Classic CSCW-Inspired Tasks

CollabSim doesn't invent arbitrary collaboration tasks—it directly borrows decades-validated experimental paradigms from CSCW:

1. Shape Factory — Resource coordination. Two agents each produce different block shapes and must trade to compose target patterns. Key challenge: trading has costs and information is asymmetric—you know what you have, but not what your partner needs.

2. DayTrader — Social dilemma. Multiple agents trade in a market, each with both an individual interest (maximize own returns) and a collective interest (maintain market stability). A multi-agent variant of the classic prisoner's dilemma.

3. Hidden Profile — Information pooling. Each agent holds only partial information; the optimal decision requires combining everyone's knowledge. Key challenge: people tend to discuss only shared information and neglect unique information—the well-known "hidden profile effect."

4. Map Task — Referential grounding. One agent sees a map, the other doesn't, and must be guided verbally to draw a route. Key challenge: conveying spatial information through language alone requires continuously checking mutual understanding—the classic scenario for building "common ground."

Configurable Interaction Conditions

One of CollabSim's core innovations is controlled manipulation of interaction conditions. Researchers can precisely adjust:

• Communication bandwidth: how much agents can send per turn (from free text to constrained formats)
• Information visibility: how much global state each agent sees (from fully public to fully private)
• Team size: 2-agent vs. 4-agent teams

Like controlled variables in psychology experiments, you can isolate the effect of, say, reducing communication bandwidth without confounding factors.

Probing Module: Peering into the Agent's Mind

CollabSim's most elegant design is the Probing Module. After each agent action, the system asks each agent:

• What is your understanding of the current task state?
• What do you think your teammates' intentions are?
• Why did you take that action?

It's like a "collaboration psychology test" for agents—not just what they did, but what they were thinking. Analyzing consistency between probe answers and actual behavior pinpoints the specific causes of collaboration failure.

Experimental Findings

The researchers tested CollabSim on four LLMs: Qwen3.6-35B-A3B, Llama-4-Maverick-17B-128E, GPT-5.5, and Claude 4.6 Sonnet.

Finding 1: Reducing communication bandwidth lowers willingness to cooperate. When agents can't express themselves freely, information transfer drops—and the willingness to cooperate itself declines. Like human teams, poor communication leads people to just "work on their own."

Finding 2: Increasing information visibility raises engagement. Agents participate more actively when they can see more of the global state. But too much information can also scatter attention—the effect is task-dependent.

Finding 3: Team size is a double-edged sword. Bigger teams mean more information sources but heavier coordination burdens. In some tasks, 4-agent teams outperform 2-agent teams; in others, the reverse.

Finding 4: Closed-source models are generally stronger at collaboration, but gaps vary by task. GPT-5.5 and Claude 4.6 Sonnet outperform open-source models on most tasks, though the gap narrows on certain collaboration dimensions.

Finding 5: Collaboration-theory-guided agent design beats simple personas. Agent prompts grounded in CSCW theory (emphasizing common ground, information sharing, and conflict repair) are more effective than simple role personas.

Qualitative Analysis: Three Failure Modes

Analyzing agent dialogues and probe responses, the researchers identified three typical collaboration failure modes:

1. Failure to coordinate around a shared goal. Agents pursue their own sub-goals and lose sight of the team objective—e.g., in Shape Factory, one agent keeps producing shapes it's good at, ignoring what its partner needs.

2. Failure to balance individual and collective interests. In DayTrader, agents know cooperation benefits the group but can't resist maximizing personal gain—mirroring human behavior in social dilemmas.

3. Failure to establish task-relevant common ground. In Map Task, the describer believes they were clear while the receiver misunderstood entirely. Agents lack the metacognitive ability to "confirm understanding."

The Bigger Picture

CollabSim's significance: it shifts multi-agent evaluation from "outcome-oriented" to "process-oriented." Task completion alone never tells you why a team failed; diagnosing the collaboration process points to concrete improvements.

This echoes the observability philosophy in software engineering—without good monitoring and diagnostic tools, you can't understand or improve complex systems. CollabSim is an "observability tool" for multi-agent systems.

For practical AI agent deployment, the work offers actionable advice: when assembling a multi-agent team, don't just pick the strongest individuals—consider their collaborative ability. Designing better collaboration protocols (e.g., periodic state synchronization, explicit confirmation of understanding) may be more effective than swapping in a stronger model.

---

Paper: CollabSim: A CSCW-Grounded Methodology for Investigating Collaborative Competence of LLM Agents

Code: github.com/neuhai/CollabSim

Authors: Jiaju Chen, Bo Sun, Yuxuan Lu, Yun Wang, Dakuo Wang, Bingsheng Yao (Northeastern University, Microsoft)