Douglas Flora, Executive Medical Director of Yung Family Cancer Center at St. Elizabeth Healthcare, President-Elect of the Association of Cancer Care Centers, and Editor-in-Chief of AI in Precision Oncology, shared a post on LinkedIn:
“It Passed the Medical Boards. Then It Miscounted the R’s in “Strawberry.”
‘The test of a first-rate intelligence is the ability to hold two opposed ideas in the mind at the same time, and still retain the ability to function.’
– F. Scott Fitzgerald
For a while, the surest way to confuse a state-of-the-art large language model was to ask it to count the r’s in the word “strawberry.” A system capable of clearing the U.S. medical licensing exam with a ninety-first percentile score would look at the word and confidently answer two.
This specific failure reveals exactly how these emerging tools operate. They do not read letters as we do; they process the world in fragments called tokens. Counting letters is a task they were never built to answer. Understanding this fundamental difference in perception is the first step toward using these platforms safely in our clinics.
In my latest essay, I break down the mechanics of AI in medicine for the working clinician. We explore how retrieval-augmented generation (RAG) keeps chatbots grounded in our actual institutional protocols, moving them away from hallucination and toward safe utility. We also look at what it means now that these systems have become agentic, transitioning from simply answering questions to actively working in loops to draft notes or match oncology patients with life-saving clinical trials.
To drive smarter cancer care, healthcare leaders must hold two things at once: a genuine wonder for the capabilities of these tools and a healthy doubt about their limitations. We need the optimism to adopt systems that can reduce the administrative burden on our colleagues, alongside the caution to remember that a model is only a reflection of its training data. We keep one foot on the dock and the other in the canoe.
A machine can absorb the collective medical knowledge of humanity, but it still has no true idea what it is looking at. That is why the most critical component of any new medical tool will always be the doctor sitting beside it. I hope you’ll find this one interested and illuminating!”
It Passed the Medical Boards. Then It Miscounted the R’s in “Strawberry.”
A working guide to how large language models and AI agents actually work, and why the clinicians who can ‘hold both’ wonder and doubt are the ones our patients will need most.
For a while, the surest way to embarrass a state-of-the-art AI was to ask how many times the letter r appears in the word strawberry. A model that had just scored in the ninety-first percentile on the U.S. medical licensing exam would study the word and answer two. There are three.
That particular failure has mostly been patched. The mechanism behind it still sits inside every one of these models, and understanding it will help you understand these emerging tools better. These machines are, by every test we used to trust, brilliant. They also do not experience the world the way we do. A good clinician has to ‘hold both’ of those at once, because that steadiness is what our patients need and expect from us. So let me show you how the machines actually work.
How a language model works
A language model does not read letters. When it trained, it broke the written world into fragments called tokens, whole common words, and split-up rare ones, and it only ever sees the fragments. “Strawberry” arrives as two or three chunks, never as ten letters, so counting the r’s was a question it was not built to answer. The model perceives a world built from different pieces than yours.
What it does with those pieces is the trick. A large language model has one core skill, learned by practicing it across a huge share of everything people have written: predict the next fragment. Given what came before, what token most plausibly comes next? Do that billions of times at an enormous scale, and something surprising falls out. The model absorbs grammar, cadence, the shape of an argument, and a startling amount of what people collectively know, all as a byproduct of getting good at the next-word guess. By 2024, that was enough for Google’s Med-Gemini to answer ninety-one percent of the medical licensing question bank correctly, past GPT-4 and past the mark a human has to clear. Knowledge that takes a physician years to accumulate now sits inside a machine that can’t count R’s.
This can cause some scary problems for those of us looking for help in our clinics. When one of these models does not know something, it rarely says so. It fills the gap with the most plausible words it can find, in the same confident voice it uses when it is right, like the overconfident Intern. Ask it for a citation, and it may hand you a study that was never written, by authors who never existed, in a journal issue that was never printed, all in clean academic prose. The field calls this hallucination. Even AlphaFold 3, whose predecessor won Demis Hassabis and John Jumper a Nobel Prize, warns in its own paper that it will sometimes generate a confident, tidy structure for a molecule that in reality has no fixed shape.
The reassuring part is that we know how to rein in the hallucinations, and these are becoming much less common in the newer models. It starts with the way a model represents meaning. Alongside predicting fragments, it turns each one into a long list of numbers, a set of coordinates, and places it in a space of thousands of dimensions where things that mean similar things sit near one another. A Google team showed the cleanest version of this in 2013. Take the coordinates for the word king, subtract man, add woman, and you land almost exactly on queen. Meaning becomes a location you can measure.
That geometry is what makes retrieval possible, and retrieval is the single most important idea for using these tools responsibly in medicine. Retrieval-augmented generation (RAG) works like this. Before the model answers, the system uses those coordinates to pull the handful of genuinely relevant documents- the current guideline, your institution’s own protocol, the real trial eligibility criteria- and sets them in front of the model with an instruction to answer from these and not from memory. The model stops free-associating and starts working from sources you chose. Retrieval on its own does not make an ungrounded chatbot safe, but a grounded system is another thing entirely, and it is the reason a language model can come to the bedside at all.
What “agentic” means
For most of the last two years, these systems answered a question and stopped. The change now underway, the one you actually missed if you stepped back in 2023, is that they have learned to act. An agent is a language model handed three things: a goal, a set of tools it is allowed to use, and permission to work in a loop. It plans a step, takes it, looks at what came back, and chooses the next step, again and again, until the goal is met. The tools might be a database search, a read of the chart, or a call to another program. The loop is what lets it act, not just answer.
You have almost certainly worked beside one already. The ambient scribe that drafts your note while you talk to the patient is an agent, and it runs in more than 600 health systems. At Mass General Brigham, measured burnout fell about a fifth in the first three months of use. A more ambitious agent, called David, sits inside Northwestern’s electronic record, reads across a patient’s notes and pathology and sequencing, and surfaces a clinical trial the patient qualifies for before the fellow has finished the note. My friend, (and AI in Precision Oncology Editorial Board member), Ezra Cohen, the oncologist who leads oncology for Tempus, has famously said: tissue is scarce, insight should not be. Trial matching used to be a clerical marathon that missed most of the people it should have caught. A patient who would once have been overlooked now has a good chance of being seen.
An agent’s autonomy is the source of both the excitement and the worry. A system that can take a step on its own can also take the wrong one, which is why the questions that matter about any agent are what it is permitted to do without asking and who is accountable when it errs. The answer to that second question has to be a person. In oncology, that person has gotta be us, guys.
Holding both
If you’ve read my stuff for some time now, you know I am an AI optimist, but I push daily for RESPONSIBLE and thoughtful adoption. We can be excited about the possibility and still sober about the risk. The biggest change under all of this is that we no longer build a narrow tool for each task. We build a foundation model that is trained once on the whole domain and then adapted cheaply to many jobs. Paige was originally trained one on more than a million pathology slides from Memorial Sloan Kettering and named it Virchow, after the physician who founded cellular pathology in the first place. Where an older program found a single cancer, Virchow finds sixteen, and it reads the rare ones better than the specialized tools built to hunt them one at a time. AlphaFold did the same for protein structure, putting a capability once reserved for a few labs into the hands of more than two million researchers. The seven-in-a-career tumor, too rare to ever train a model of its own, comes within reach because the model already learned the domain. This is genuinely new and is really good news for our patients.
The optimism only holds up if we stay honest about where these tools can still fail. A model is a portrait of the data it learned from, and where that data is thin, the portrait misleads quietly until a patient is harmed. A skin-cancer classifier trained mostly on pale skin misses melanoma on dark skin. A pulse oximeter that reads falsely high on darker skin can place that error straight into the chart, where the next model treats the bad number as truth. Genomic databases built mostly from people of European descent yield poorer results for everyone else. Each of those failures is why a clinician belongs in the loop, and why the questions worth asking of any tool are plain enough to ask without a data-science degree: on patients it never saw, how did it perform, and for whom, and who answers when it is wrong. A vendor who cannot tell you where a tool fails has not looked hard enough to sell it to you.
I am an optimist about this, and I try to earn the optimism rather than assume it. The excitement and the caution are the two hands a good clinician has always worked with at once, the one reaching for what might help, the other making sure it will not harm. I liken this one to keeping one foot on the dock and the other in the canoe. Our patients need both innovation andthoughtful caution. So do our colleagues, the ones worn thin by the chart and tempted either to wave all of this away or to swallow it whole, who need to watch a few of us do it well first.
Start with the strawberry. A thing can be astonishing and still have no idea what it is looking at. That is why it needs a good doctor beside it.
Other articles about AI on OncoDaily.