This week, the independent clinical AI research group ARISE released NOHARM v2, a study evaluating whether AI recommendations are safe in realistic clinical scenarios. The group is led by physicians at Stanford and Harvard.
Doximity Ask received the highest score in real-world testing of commercially available products by physicians (75.8%), ahead of ChatGPT Health (68.4%) and OpenEvidence (60.8%).
Unlike traditional medical AI benchmarks, NOHARM evaluates whether a model's recommendations could contribute to patient harm rather than performance on multiple-choice questions. The benchmark was developed by more than 50 researchers, including 29 board-certified physicians, and included 1,100 case scenarios.
The NOHARM study found medical AI systems performed well with prompts as they arise naturally in clinical practice. General-purpose models like ChatGPT or Gemini often improved with carefully engineered prompts, but even then did not reach the performance of the leading medical systems. Doximity Ask is intended to provide reliable guidance without requiring specialized prompting.
The study also evaluated whether models gave consistent guidance when the underlying clinical problem was unchanged. To test this, researchers made small, clinically irrelevant edits to patient scenarios — for example, changing a patient's age from 71 to 73 — and looked for changes in outputs. General-purpose models were substantially more likely to vary their answers. Specialized systems like Doximity Ask were much less sensitive to these changes. If the medical facts haven't changed, the system's conclusions shouldn't either.
Beyond the rankings, the study consistently found that systems designed specifically for clinical care outperformed general-purpose AI across multiple evaluations. The findings suggest that clinical performance requires more than general language ability. Systems must also produce recommendations that are stable and grounded in the underlying evidence.
Every system in this study, ours included, had measurable gaps. The right response is to measure them, make them visible, and keep improving — which is what the 11,000+ physicians in our PeerCheck™ program help us do every day.
We thank the ARISE investigators for raising the standard on how clinical AI is evaluated. Independent, safety-focused evaluation benefits physicians, health systems, and ultimately patients.