Microsoft’s diagnostic AI scored higher than generalist physicians on two written case tests in a peer-reviewed study published in Science on October 22. The results do not show how the system would perform with patients.

The Microsoft AI Diagnostic Orchestrator, known as MAI-DxO, worked through cases by requesting information and tests, then updating its assessment. Microsoft says the approach was designed to test clinical reasoning step by step, not just whether the final diagnosis was right.

How the scores compared

On 304 complex cases drawn from The New England Journal of Medicine, MAI-DxO reached 83.2% accuracy. Generalist physicians using reference tools scored 36.1%, according to Medical Economics’ account of the study.

On 325 emergency department cases, the system scored 80.3%, compared with 46.8% for physicians. Microsoft described the results and its planned next steps.

  • Complex cases: 83.2% for MAI-DxO, compared with 36.1% for generalist physicians.
  • Emergency department cases: 80.3% for MAI-DxO, compared with 46.8% for physicians.

What the tests did not show

The cases were presented as text. They did not include the full range of information clinicians may use, such as imaging. The comparison was with individual generalist physicians, not specialists or medical teams.

The study measured diagnostic accuracy. It did not test whether MAI-DxO would recommend safe treatment or improve patients’ health. Strong scores on written cases are not evidence that the system can diagnose people safely in routine care.

A patient study is planned

Microsoft says it plans a prospective study with Beth Israel Deaconess Medical Center in Boston. Patients will interact with the system, and researchers will compare its performance with physicians’ independent assessments. The study is planned, not completed.

For now, MAI-DxO has performed strongly on written medical cases. Whether it can help clinicians care for patients safely remains unanswered.