Roupen Odabashian, Oncologist at Abbotsford Regional Hospital and Cancer Centre, Founder at MeDucation AI, Podcast Host at OncoDaily, shared on LinkedIn:
”A new Stanford and Harvard State of Clinical AI report found that leading models lose more than a third of their accuracy the moment a case stops being clean.
Give a model a tidy vignette with every fact laid out and it performs well. Make it ask follow up questions, handle missing information, or revise its thinking as new details arrive, and it drops toward medical student level on reasoning under uncertainty, per work published in NEJM AI.
This matches what I see in clinic. Real oncology is almost never the clean vignette. The patient forgets a med, the path report is pending, the story changes on the third question. The hard part of medicine is not producing the answer once you have all the data. It is knowing what to ask when you do not.
So the benchmark that matters is not accuracy on a finished case. It is accuracy while the case is still messy and incomplete. Almost none of our published numbers measure that.
Until they do, I would be careful reading a high board exam score as evidence that a model is ready for the room.”
Read more articles about Harvard on OncoDaily.