Microsoft’s AI Diagnostic System Outpaces Physicians in Complex Case Simulations
A new study reveals that Microsoft’s multi-agent AI system correctly diagnosed difficult medical scenarios at a rate more than double that of generalist doctors.
By Rafael Collins ·
For patients facing difficult medical diagnoses, a new benchmark suggests that artificial intelligence is rapidly closing the gap with human expertise, particularly in scenarios where generalist physicians struggle. A study published in the journal Science this week indicates that a new diagnostic system developed by Microsoft can identify complex medical conditions with significantly higher accuracy than individual doctors working alone. While this does not replace the specialist teams that typically handle such cases in a real-world hospital, the results highlight a shift in how medical reasoning might be supported in the near future.
The performance metrics are stark. In simulations using complex cases from the New England Journal of Medicine, the system, known as MAI-DxO, achieved a correct diagnosis rate of 83%. By comparison, primary care physicians and general internists, working individually with standard reference materials, were correct only 36% of the time. The study notes that these specific cases are usually managed by specialists or teams in practice, which were not included in this direct comparison. However, the data points to a potential efficiency gain in triage and initial assessment for ambiguous symptoms.
MAI-DxO operates by coordinating a panel of AI agents that mimic a medical board. These agents question the patient, request specific tests, and debate potential diagnoses before arriving at a conclusion. Microsoft tested the system using models from various providers, including OpenAI, Google, Anthropic, Meta, and DeepSeek. The most accurate results were generated using OpenAI’s GPT-5 and GPT-5.5 models. When using GPT-5.5, a model released in April 2026, the orchestration system boosted accuracy from 72% to nearly 87% on complex cases, while also slightly reducing the cost per case.
Cost and efficiency were also measured in the study. Running GPT-5 alone on complex cases cost approximately $8,400 per case with 78.9% accuracy. MAI-DxO reduced that cost by about $3,000 while improving accuracy by roughly four points. However, on emergency department cases, the system provided little improvement in either cost or accuracy compared to using the model alone. The study suggests that the orchestration layer is most beneficial for complex, ambiguous cases and when using less capable underlying models, such as Gemini 2.5 Flash, which improved from 46.4% accuracy to nearly 69% with the system’s assistance.
Mustafa Suleyman, CEO of Microsoft AI, described the findings as a step toward "medical superintelligence," a term defined as surpassing current human-led health systems in reasoning and coordination. Suleyman, who joined Microsoft in 2024 to lead its AI efforts, stated that the company is collaborating with the Mayo Clinic to train a foundational healthcare model scheduled for release next year. He predicts that by 2027, the technology will enable "streaming highly accurate predictions" about how medical records might unfold, a capability he describes as "gold dust" for clinicians.
The study positions Microsoft alongside Google and OpenAI in the race to build AI that matches or exceeds human diagnostic capability. Microsoft is moving beyond simple benchmarks, such as answering medical exam questions, toward integrating AI directly into clinical workflows. A spokesperson for the company noted that while accuracy gains shrink as models approach theoretical maximums, the orchestration system helps optimize both accuracy and cost. Suleyman believes the progress will accelerate rapidly once the systems are in production, creating a "flywheel" of real-world feedback that identifies gaps and improves the models.
To validate these capabilities outside of simulations, Microsoft has announced a partnership with Beth Israel Deaconess Medical Center in Boston. The collaboration will evaluate the clinical reasoning of MAI-DxO across various hospital settings, including emergency and urgent care environments. Patients will receive standard treatment while consenting to parallel engagement with the AI system. The results will be compared to those of human doctors. Dominic King, vice president of Health at Microsoft AI, called this the "biggest study of its kind," noting that it will test the system against the diverse and complex issues typical of a Boston emergency department on a busy evening.
The initiative follows a similar move by Google, which published a study in The Lancet this month evaluating its AMIE diagnostic AI in real-world urgent care visits. Microsoft views this expansion into clinical decision support as a natural evolution of its existing tools for dictation and radiology. Suleyman emphasized that the company aims to hold the "absolute number one state-of-the-art" position in medical AI over the next few years, arguing that this focus on healthcare differentiation sets Microsoft apart from other frontier labs.
