AI panel outperforms solo doctors in complex medical simulations

83 percent. That is the accuracy rate achieved by Microsoft’s new AI diagnostic system, MAI-DxO, when tackling complex medical cases in a recent simulation. By comparison, primary care doctors and general internists, working individually with reference materials, were correct only 36 percent of the time. These results, published Thursday in the journal Science, have prompted Microsoft AI CEO Mustafa Suleyman to declare a significant step toward what he calls “medical superintelligence.”

The system does not function as a single, monolithic brain. Instead, MAI-DxO coordinates multiple AI agents that operate like a virtual panel of physicians. These agents collectively interrogate patient data, order necessary tests, and debate potential diagnoses. While the headline number is striking, it is important to note that the comparison was made against generalists. In real-world scenarios, cases of this complexity are typically managed by specialists or multidisciplinary teams, who were not included in the benchmark.

The study reveals that the bulk of the performance improvement stems from the underlying power of the large language models used, rather than the orchestration system itself. Microsoft tested MAI-DxO with models from OpenAI, Google, Anthropic, Meta, and DeepSeek. The highest results were achieved using OpenAI’s GPT-5 and GPT-5.5. When using GPT-5 alone with a basic prompt, the system scored 78.9 percent on complex cases at a cost of approximately $8,400 per case. The MAI-DxO orchestration reduced this cost by about $3,000 but increased accuracy by only four percentage points.

The benefits of the orchestration layer become more pronounced with less capable models. For instance, Gemini 2.5 Flash scored 46.4 percent on its own but rose to nearly 69 percent when assisted by MAI-DxO. However, the gap between weaker and stronger models remained larger than the boost provided by the coordination system. Furthermore, on emergency department cases, there was little to no improvement in accuracy or cost between the standalone model and the assisted version.

Dominic King, vice president of Health at Microsoft AI, described the findings as “very substantial and exciting.” He highlighted the growing mismatch between the demand for medical expertise and its supply as the primary challenge in healthcare delivery. By publishing these results, Microsoft joins Google and OpenAI in asserting that its AI can match or exceed human performance in diagnostic tasks. The company is also working with the Mayo Clinic to train a foundation model specifically for healthcare, with a release planned for next year.

Suleyman, who previously co-founded DeepMind, believes the next 24 to 36 months will see an “order of magnitude” improvement in healthcare models. He anticipates a “coding style breakthrough” for medical records by 2027, allowing for highly accurate predictions of patient outcomes. This foresight, he argues, is invaluable for clinicians. To test these capabilities in a real-world setting, Microsoft has partnered with Beth Israel Deaconess Medical Center in Boston.

In this upcoming evaluation, patients will receive standard treatment, including triage and specialist consultations. Those who consent will also interact with MAI-DxO in parallel. Researchers will then compare the AI’s recommendations against those of the human physicians. King noted that this will be one of the largest studies of its kind, covering both emergency and urgent care environments. He emphasized that the technology is ready for such rigorous testing, particularly given the complexity of cases seen in a busy Boston emergency department on a Friday evening.

Microsoft’s strategy extends beyond single-task benchmarks like answering exam questions. The company aims to integrate AI into the broader arc of care, working alongside clinicians and healthcare systems. Suleyman stated that this focus on medical superintelligence could serve as a differentiator for Microsoft compared to other frontier labs. The company already sells dictation and radiology tools to clinicians and views MAI-DxO as a natural expansion of these efforts, aiming to provide the “absolute number one” state-of-the-art clinical decision support products in the coming years.