AAAI-27 Workshop · Third edition of WMAC
Measuring Collective Intelligence in Agentic Systems
- February 22–23, 2027 (one-day workshop; exact day TBD)
- Montréal, Canada · co-located with AAAI-27
–days
–hours
–min
Overview
The measurement gap
Modern agent deployments rarely consist of a single model. A coding assistant delegates to sub-agents for retrieval, editing, and verification; research prototypes organize dozens of agents into societies that negotiate and trade; and most real systems keep humans in the loop. How well these pieces work together decides how well the system works.
Our evaluation tools have not kept up. Standard benchmarks score one model on one task and say little about coordination overhead, specialization, communication efficiency, or whether adding another agent helps at all. Failure analyses of multi-agent systems point the same way: recurring problems in specification, coordination, and verification are not surfaced by task-level scores.
The first two WMAC editions asked how to make LLM-based agents collaborate. In their discussion sessions, one question kept coming back and stayed open: how do we know a group of agents is actually working well together? This edition takes that question as its topic.
Measurement as the theme
The first two editions asked how to make agents collaborate. This one asks how to tell whether they do.
Breakout discussions
Small groups around posted questions, added because past open-discussion sessions overflowed.
Live industry demos & panel
Deployed multi-agent systems shown live, plus the panel that drew the liveliest exchanges in both prior editions.
Call for papers
Topics of interest
Submissions on the following topics are welcome, but not limited to them.
Metrics & evaluation methodologies
New metrics for coordination quality, communication efficiency, specialization, and robustness in agent teams. Extending single-agent frameworks such as AgentBoard, and multi-agent arenas such as LLMArena, to measure collective behavior.
Benchmarks, environments & simulations
Controlled environments where collective behavior can be studied reproducibly, including negotiation and social settings, and benchmarks that score collaboration and competition directly.
Foundations & theories
Formal definitions and scaling laws for collective intelligence, with connections to how it has been studied in biological and economic systems.
Hierarchical & sub-agent architectures
Measurement questions specific to planner–executor and manager–worker structures, where performance is attributable to no single component.
Human–agent collectives
Evaluation of mixed human–AI teams and collective decision making.
Applications
Case studies from software engineering, scientific discovery, robotics, healthcare, and economics where collective performance was measured, successfully or not.
Invited speakers & panelists
Invited participants Tentative
This list is tentative and pending final confirmation of availability. Four will give invited talks. The rest join the panel and help moderate the breakout discussions. Talk assignments will be announced later.

Bo An
Nanyang Technological University
Game-theoretic and reinforcement-learning approaches to multi-agent decision making.

Joel Z. Leibo
Google DeepMind · King's College London
What Melting Pot and Concordia reveal about multi-agent cooperation, and what they still miss.

Zhijing Jin
University of Toronto · Vector Institute
Causal reasoning with LLMs, and what GovSim and MoralSim show about cooperation and norms in societies of LLM agents.

Samy Bengio
Apple · EPFL
How capability scales as AI systems grow more composite.

Anita Williams Woolley
Carnegie Mellon University
Creator of the “c factor,” the canonical measure of collective intelligence in groups.

John Horton
MIT · Panelist
LLM agents as economic agents: markets, incentives and what economics says about measuring them.

AJung Moon
McGill University · Mila · Panelist
Human-robot interaction and AI ethics: how AI shapes people’s decisions, and how to measure it.
Program
Proposed schedule
A full day built for interaction. The final agenda will be aligned with the AAAI-27 conference timetable.
Small groups around posted questions: what to measure, how to compare systems, how to evaluate human–agent teams. Each group reports back.
Live walkthroughs of deployed multi-agent systems, with questions mid-demo.
- Invited talk
- Contributed
- Poster
- Interactive
- Panel
Timeline
Important dates
Dates follow the schedule set by the AAAI-27 workshop chairs and may change.
Submission
Submission guidelines
Full papers
up to 8 pages
Original research on measuring collective intelligence in agentic systems.
Short, demo & position papers
up to 4 pages
Early results, system demos, case studies, and position statements.
Page counts exclude references and appendices. All papers use the AAAI-27 format.
- Format: AAAI-27. Use the official AAAI-27 Author Kit, anonymized for review.
- Page limits cover the main text only. References and appendices do not count and are unlimited. Reviewers are not required to read the appendix, so the main paper should stand on its own.
- Non-archival. Presenting here does not prevent publishing elsewhere.
- Double-blind review on OpenReview, with three reviews per paper.
- Best Student Paper Award for master's and PhD students.
- Expected: 40–50 submissions, roughly 15–20 accepted. Three oral talks, the rest posters, each with a short lightning pitch.
- Industry demos welcome. Presenters should prepare a recorded fallback.
Team
Organizers
Four of the five organizers ran previous WMAC editions. Shu chaired both, Geramifard and Marin co-organized both, and Zhang co-chaired 2026.

Raphael Shu
Salesforce · OpenAgents · Workshop chair
Principal Scientist working on AI agents. PhD, University of Tokyo. Chaired WMAC 2025 and 2026.

Alborz Geramifard
Distinguished Scientist working on agentic RL. Previously led Conversational AI at Meta and Amazon. PhD, MIT.

Alex Marin
Thomson Reuters Labs
Lead Applied Scientist, previously at Microsoft. PhD, University of Washington, where he is Affiliate Professor.

Yusen Zhang
Columbia University
Postdoctoral Research Scientist working on LLM agents and multi-agent systems. Co-chaired WMAC 2026.

Asli Celikyilmaz
Microsoft · University of Washington
Agentic AI Research Director and Affiliate Faculty. IEEE Fellow; Editor-in-Chief of TACL.
The series
Third edition of WMAC
The Workshop on Multi-Agent Collaboration at AAAI has drawn a growing community since 2025.
WMAC 2025
33submissions
178registered
WMAC 2026
42submissions
360registered
WMAC 2027
40–50expected submissions

Questions?
Contact the workshop chair Raphael Shu.
