Collaborative Intelligence Benchmark tests 18 AI models as teammates; best scores 82.2/100
A paper introduces the Collaborative Intelligence Benchmark, built to measure how well an AI agent works as a teammate. The authors start from the observation that AI progress is usually judged by what a single model can do alone, while teams of AI agents are starting to make breakthroughs and discoveries in science, medicine, mathematics and software engineering. As agents work in teams with other agents and people, the ability to collaborate becomes a core part of their intelligence. Research on human teams has measured this ability for decades, and the benchmark is inspired by that work.
The benchmark has 240 tasks across 8 dimensions of teamwork. The examples given range from refusing an unsafe shortcut to checking the work that a weaker model leaves behind. Each task comes from a real record or an existing benchmark, from aviation accident reports and clinical cases to coding tasks and negotiations, and each has a programmatically verifiable outcome. To separate teamwork from individual ability, models are scored only on tasks they can solve alone when they have full information.
The authors evaluate 18 closed- and open-weight models. Their main finding is that every model, across sizes and families, has clear weaknesses as a teammate. The best model scores only 82.2 out of 100.
The failures are specific. Frontier models almost always refuse unsafe shortcuts and report misconduct, yet they cave under interpersonal pressure. In negotiation tasks they reach the best overall deal in only 14% of conflicts. In another scenario, their pass rate drops from 97% to 81% when they take over a task in which a weaker model has cheated.
The authors also stress that general capability does not guarantee good teamwork: models with nearly the same capability score differ by up to 11 points as teammates.
The benchmark includes 160 public tasks with code and data, plus 80 held-out tasks for evaluation. The authors close by saying that as AI agents increasingly work alongside other agents and people, the ability to collaborate effectively is becoming a crucial capability to measure and develop.
Key facts
- The Collaborative Intelligence Benchmark has 240 tasks across 8 dimensions of teamwork, drawn from real records or existing benchmarks such as aviation accident reports, clinical cases, coding tasks and negotiations.
- Of 18 closed- and open-weight models evaluated, the best scored only 82.2 out of 100, and every model showed clear weaknesses as a teammate.
- Frontier models almost always refuse unsafe shortcuts and report misconduct, but cave under interpersonal pressure: they reach the best overall deal in only 14% of conflicts.
- Frontier models' pass rate falls from 97% to 81% when they take over a task in which a weaker model has cheated.
- Models with nearly the same capability score differ by up to 11 points as teammates; 160 tasks are public with code and data, and 80 are held out.
Why it matters
Most AI benchmarks test what one model can do alone. This one targets a different question: how well an agent behaves as part of a team of agents and people. The authors say teams of AI agents are already starting to make breakthroughs in science, medicine, mathematics and software engineering, so collaboration is becoming part of what intelligence means for these systems. Their central result is that general capability does not guarantee good teamwork: models with nearly the same capability score differ by up to 11 points as teammates.
Who it affects
The findings bear on anyone building or deploying multi-agent systems, or putting agents alongside human colleagues. They also bear on model developers, since the paper reports clear teamwork weaknesses in every one of the 18 models tested, across sizes and families, closed and open weights alike.
How to use it
The benchmark ships 160 public tasks with code and data, which teams can use to probe their own models on teamwork behaviours such as refusing an unsafe shortcut or checking a weaker model's work. A further 80 tasks are held out for evaluation. Tasks have programmatically verifiable outcomes, so scoring does not depend on subjective judging.
How solid is it
The design has some careful touches. Tasks come from real records or existing benchmarks, outcomes are programmatically verifiable, and models are scored only on tasks they can solve alone with full information, which separates teamwork from raw ability. The headline numbers are the authors' own results. The source does not name the models tested, so the reader cannot tell which model scored 82.2 or which count as frontier. It also does not say how the overall score out of 100 is aggregated.
Risks and caveats
The figures come from the paper's own benchmark and should be read as the authors' findings, not independent replication. The source describes only two of the 8 teamwork dimensions, so the full coverage of the 240 tasks is not visible from this text. Because the best score is 82.2 out of 100 and the score's aggregation is not explained, it is hard to judge how severe that gap is in practice.