Skip to main content

AAAI-27 Workshop · Third edition of WMAC

Measuring Collective Intelligence in Agentic Systems

  • February 22–23, 2027 (one-day workshop; exact day TBD)
  • Montréal, Canada · co-located with AAAI-27
Palais des congrès de Montréal · AAAI-27 venue
Submissions dueNovember 20, 2026 · 23:59 AoE

–days

–hours

–min

Overview

The measurement gap

Modern agent deployments rarely consist of a single model. A coding assistant delegates to sub-agents for retrieval, editing, and verification; research prototypes organize dozens of agents into societies that negotiate and trade; and most real systems keep humans in the loop. How well these pieces work together decides how well the system works.

Our evaluation tools have not kept up. Standard benchmarks score one model on one task and say little about coordination overhead, specialization, communication efficiency, or whether adding another agent helps at all. Failure analyses of multi-agent systems point the same way: recurring problems in specification, coordination, and verification are not surfaced by task-level scores.

The first two WMAC editions asked how to make LLM-based agents collaborate. In their discussion sessions, one question kept coming back and stayed open: how do we know a group of agents is actually working well together? This edition takes that question as its topic.

Machine learningMulti-agent systemsReinforcement learningEconomicsCognitive science
New this year

Measurement as the theme

The first two editions asked how to make agents collaborate. This one asks how to tell whether they do.

New this year

Breakout discussions

Small groups around posted questions, added because past open-discussion sessions overflowed.

Returning

Live industry demos & panel

Deployed multi-agent systems shown live, plus the panel that drew the liveliest exchanges in both prior editions.

Call for papers

Topics of interest

Submissions on the following topics are welcome, but not limited to them.

📏

Metrics & evaluation methodologies

New metrics for coordination quality, communication efficiency, specialization, and robustness in agent teams. Extending single-agent frameworks such as AgentBoard, and multi-agent arenas such as LLMArena, to measure collective behavior.

🧪

Benchmarks, environments & simulations

Controlled environments where collective behavior can be studied reproducibly, including negotiation and social settings, and benchmarks that score collaboration and competition directly.

∑

Foundations & theories

Formal definitions and scaling laws for collective intelligence, with connections to how it has been studied in biological and economic systems.

🌳

Hierarchical & sub-agent architectures

Measurement questions specific to planner–executor and manager–worker structures, where performance is attributable to no single component.

🤝

Human–agent collectives

Evaluation of mixed human–AI teams and collective decision making.

🚀

Applications

Case studies from software engineering, scientific discovery, robotics, healthcare, and economics where collective performance was measured, successfully or not.

Invited speakers & panelists

Invited participants Tentative

This list is tentative and pending final confirmation of availability. Four will give invited talks. The rest join the panel and help moderate the breakout discussions. Talk assignments will be announced later.

Bo An

Bo An

Nanyang Technological University

Game-theoretic and reinforcement-learning approaches to multi-agent decision making.

Joel Z. Leibo

Joel Z. Leibo

Google DeepMind · King's College London

What Melting Pot and Concordia reveal about multi-agent cooperation, and what they still miss.

Zhijing Jin

Zhijing Jin

University of Toronto · Vector Institute

Causal reasoning with LLMs, and what GovSim and MoralSim show about cooperation and norms in societies of LLM agents.

Samy Bengio

Samy Bengio

Apple · EPFL

How capability scales as AI systems grow more composite.

Anita Williams Woolley

Anita Williams Woolley

Carnegie Mellon University

Creator of the “c factor,” the canonical measure of collective intelligence in groups.

John Horton

John Horton

MIT · Panelist

LLM agents as economic agents: markets, incentives and what economics says about measuring them.

AJung Moon

AJung Moon

McGill University · Mila · Panelist

Human-robot interaction and AI ethics: how AI shapes people’s decisions, and how to measure it.

Program

Proposed schedule

A full day built for interaction. The final agenda will be aligned with the AAAI-27 conference timetable.

Opening remarks5 min
Invited talk 145 min
Oral presentation 120 min
Coffee break20 min
Invited talk 245 min
Lightning talks15 min
Poster session60 min
Lunch break90 min
Breakout discussions45 min

Small groups around posted questions: what to measure, how to compare systems, how to evaluate human–agent teams. Each group reports back.

Industry demo session45 min

Live walkthroughs of deployed multi-agent systems, with questions mid-demo.

Coffee break20 min
Invited talk 345 min
Oral presentation 220 min
Invited talk 445 min
Oral presentation 320 min
Panel discussion30 min
Awards & closing10 min
  • Invited talk
  • Contributed
  • Poster
  • Interactive
  • Panel

Timeline

Important dates

Oct 2, 2026Call for participation
Nov 20, 2026Paper submission deadline (23:59 AoE)
Dec 2, 2026Notification of acceptance
Feb 22–23, 2027Workshop, Montréal

Dates follow the schedule set by the AAAI-27 workshop chairs and may change.

Submission

Submission guidelines

Full papers

up to 8 pages

Original research on measuring collective intelligence in agentic systems.

Short, demo & position papers

up to 4 pages

Early results, system demos, case studies, and position statements.

Page counts exclude references and appendices. All papers use the AAAI-27 format.

  • Format: AAAI-27. Use the official AAAI-27 Author Kit, anonymized for review.
  • Page limits cover the main text only. References and appendices do not count and are unlimited. Reviewers are not required to read the appendix, so the main paper should stand on its own.
  • Non-archival. Presenting here does not prevent publishing elsewhere.
  • Double-blind review on OpenReview, with three reviews per paper.
  • Best Student Paper Award for master's and PhD students.
  • Expected: 40–50 submissions, roughly 15–20 accepted. Three oral talks, the rest posters, each with a short lightning pitch.
  • Industry demos welcome. Presenters should prepare a recorded fallback.

Team

Organizers

Four of the five organizers ran previous WMAC editions. Shu chaired both, Geramifard and Marin co-organized both, and Zhang co-chaired 2026.

Raphael Shu

Raphael Shu

Salesforce · OpenAgents · Workshop chair

Principal Scientist working on AI agents. PhD, University of Tokyo. Chaired WMAC 2025 and 2026.

Alborz Geramifard

Alborz Geramifard

LinkedIn

Distinguished Scientist working on agentic RL. Previously led Conversational AI at Meta and Amazon. PhD, MIT.

Alex Marin

Alex Marin

Thomson Reuters Labs

Lead Applied Scientist, previously at Microsoft. PhD, University of Washington, where he is Affiliate Professor.

Yusen Zhang

Yusen Zhang

Columbia University

Postdoctoral Research Scientist working on LLM agents and multi-agent systems. Co-chaired WMAC 2026.

Asli Celikyilmaz

Asli Celikyilmaz

Microsoft · University of Washington

Agentic AI Research Director and Affiliate Faculty. IEEE Fellow; Editor-in-Chief of TACL.

The series

Third edition of WMAC

The Workshop on Multi-Agent Collaboration at AAAI has drawn a growing community since 2025.

Questions?

Contact the workshop chair Raphael Shu.

Raphael Shu