Microsoft's MAI-DxO diagnosis study is now in Science, with an open benchmark and a hospital trial planned

Microsoft AI’s diagnostic system, MAI-DxO, has been published in Science as “Sequential diagnosis with language models.” Alongside the paper, Microsoft says it has released an open benchmark and is planning a study with real patients.

What was tested

Most medical AI tests hand the model a full case and ask for an answer. SDBench, the benchmark here, starts with a short case summary. The system must ask questions and order tests one at a time, each with a cost, before committing to a diagnosis. MAI-DxO runs language models as what Microsoft calls a panel of virtual physicians, which propose diagnoses, pick questions or tests and challenge the emerging view, weighing each test’s value against its cost.

The numbers

On 304 hard cases from the New England Journal of Medicine, Microsoft reports 83% accuracy for MAI-DxO. Physicians on the same cases scored 22% without references and 36% with them. On emergency cases from the MIMIC-IV-ED dataset it reports 80% against 47% for physicians. Microsoft says the gains held when newer models, including GPT-5.5, were swapped in without changing the orchestration. These are Microsoft’s figures on a benchmark it built.

What is open

The SDBench site publishes the benchmark for a selection of NEJM cases, with the gatekeeper that answers test orders from the full case, the judge that scores the final diagnosis, and the cost-estimation code, so other teams can test their own models.

What comes next

Microsoft plans a prospective study with Beth Israel Deaconess Medical Center in which patients interact with the system directly and its performance is measured against physicians’ independent assessments. That is the test a benchmark cannot replace.

Why it matters

Diagnosis is a sequence of choices, each with a cost. A benchmark that scores the path as well as the answer is a better yardstick for medical agents than multiple-choice exams.