AAAI-27 Workshop · Third edition of WMAC
Measuring Collective Intelligence in Agentic Systems
- February 22–23, 2027 (one-day workshop; exact day TBD)
- Montréal, Canada · co-located with AAAI-27
–days
–hours
–min
Overview
The measurement gap
Modern agent deployments rarely consist of a single model. A coding assistant delegates to sub-agents for retrieval, editing, and verification; research prototypes organize dozens of agents into societies that negotiate and trade; and most real systems keep humans in the loop. How well these pieces work together decides how well the system works.
Our evaluation tools have not kept up. Standard benchmarks score one model on one task and say little about coordination overhead, specialization, communication efficiency, or whether adding another agent helps at all. Failure analyses of multi-agent systems point the same way: recurring problems in specification, coordination, and verification are not surfaced by task-level scores.
The first two WMAC editions asked how to make LLM-based agents collaborate. In their discussion sessions, one question kept coming back and stayed open: how do we know a group of agents is actually working well together? This edition takes that question as its topic.
Measurement as the theme
The first two editions asked how to make agents collaborate. This one asks how to tell whether they do.
Breakout discussions
Small groups around posted questions, added because past open-discussion sessions overflowed.
Live industry demos & panel
Deployed multi-agent systems shown live, plus the panel that drew the liveliest exchanges in both prior editions.
Call for papers
Topics of interest
Submissions on the following topics are welcome, but not limited to them.
Metrics & evaluation methodologies
New metrics for coordination quality, communication efficiency, specialization, and robustness in agent teams. Extending single-agent frameworks such as AgentBoard, and multi-agent arenas such as LLMArena, to measure collective behavior.
Benchmarks, environments & simulations
Controlled environments where collective behavior can be studied reproducibly, including negotiation and social settings, and benchmarks that score collaboration and competition directly.
Foundations & theories
Formal definitions and scaling laws for collective intelligence, with connections to how it has been studied in biological and economic systems.
Hierarchical & sub-agent architectures
Measurement questions specific to planner–executor and manager–worker structures, where performance is attributable to no single component.
Human–agent collectives
Evaluation of mixed human–AI teams and collective decision making.
Applications
Case studies from software engineering, scientific discovery, robotics, healthcare, and economics where collective performance was measured, successfully or not.
Invited speakers & panelists
Invited participants Tentative
This list is tentative and pending final confirmation of availability. Four will give invited talks. The rest join the panel and help moderate the breakout discussions. Talk assignments will be announced later.
Michael Wooldridge
University of Oxford
Author of textbooks that shaped the multi-agent systems field. On what classical MAS research already knows about measuring collective behavior.
Samy Bengio
Apple · EPFL
How capability scales as AI systems grow more composite.
Mikaela Grace
Anthropic
How deployed agents coordinate in practice, and where coordination breaks.
Bo An
Nanyang Technological University
Game-theoretic and reinforcement-learning approaches to multi-agent decision making.
Noah Goodman
Stanford University
The cognitive-science view of collective reasoning.
Program
Proposed schedule
A full day built for interaction. The final agenda will be aligned with the AAAI-27 conference timetable.
Small groups around posted questions: what to measure, how to compare systems, how to evaluate human–agent teams. Each group reports back.
Live walkthroughs of deployed multi-agent systems, with questions mid-demo.
- Invited talk
- Contributed
- Poster
- Interactive
- Panel
Timeline
Important dates
Dates follow the schedule set by the AAAI-27 workshop chairs and may change.
Submission
Submission guidelines
Full papers
up to 8 pages
Original research on measuring collective intelligence in agentic systems.
Short, demo & position papers
up to 4 pages
Early results, system demos, case studies, and position statements.
- Non-archival. Presenting here does not prevent publishing elsewhere.
- Double-blind review on OpenReview, with three reviews per paper.
- Best Student Paper Award for master's and PhD students.
- Expected: 40–50 submissions, roughly 15–20 accepted. Three oral talks, the rest posters, each with a short lightning pitch.
- Industry demos welcome. Presenters should prepare a recorded fallback.
Team
Organizers
Four of the five organizers ran previous WMAC editions. Shu chaired both, Geramifard and Marin co-organized both, and Zhang co-chaired 2026.
Raphael Shu
OpenAgents · Workshop chair
Principal Scientist working on AI agents. PhD, University of Tokyo (2020). Led Dialog2API and agentic capability development for the Titan LLM at AWS. ACL 2017 Outstanding Paper Award. Chaired WMAC 2025 and 2026.
Alborz Geramifard
Distinguished Scientist working on agentic reinforcement learning. Previously led Conversational AI teams at Meta and Amazon. PhD, MIT. Has organized more than ten workshops and summits; Area Chair for ACL, EMNLP, NAACL, and RLC.
Alex Marin
Thomson Reuters Labs
Lead Applied Scientist, previously Principal Applied Scientist at Microsoft. PhD, University of Washington, where he is Affiliate Professor in Linguistics. Workshop Chair for EMNLP and Program Chair for WeCNLP.
Yusen Zhang
Columbia University
Postdoctoral Research Scientist working on LLM agents and multi-agent systems. Co-chaired WMAC 2026. SAC Award at NAACL 2025; Best Workshop Paper at AAAI 2025.
Asli Celikyilmaz
Microsoft · University of Washington
Agentic AI Research Director and Affiliate Faculty. Research on multi-agent societies and optimization that balances individual agent objectives against collaborative outcomes. IEEE Fellow; Editor-in-Chief of TACL.
The series
Third edition of WMAC
The Workshop on Multi-Agent Collaboration at AAAI has drawn a growing community since 2025.
WMAC 2025
33submissions
178registered
WMAC 2026
42submissions
360registered
WMAC 2027
40–50expected submissions
Questions?
Contact the workshop chair Raphael Shu.