Joe Lennerz, Medical Director, Pathology Innovations at Natera, shared on LinkedIn:
“How do you regulate an AI tool when the same input does not always produce the same output? That is the fundamental challenge with LLMs and generative AI: ‘predetermined’ becomes difficult when outputs are probabilistic.
The regulatory approach was called pre-determined change control plan (PCCP). And these work, well, when changes can be anticipated, bounded, and validated in advance… but GenAI behavior can change through the model, prompts, data, guardrails, orchestration, or third-party updates.
The answer to this challenging question is simple: ‘by using Shadow’.
The FDA’s new discussion paper on generative AI-enabled medical devices, tackles exactly this problem. The emerging solution is to shift from predetermining the change to predetermining the performance boundary.
Rather than requiring every future behavior to be predictable, define what the system must ‘reliably be able to do’ and continuously test whether it remains within that validated capability envelope.
This is where ‘shadow deployment’ becomes especially powerful. Before—or as part of a (pre-clinical) soft launch, the AI can operate on real patients in the real clinical workflow while its outputs are captured but do not yet influence care. This is distinct from a sandbox… let’s briefly distinguish:
Sandbox: Can it perform?
Shadow: Does it perform here?
Postmarket/post-launch: Does it still perform?
The Shadow environment therefore creates the bridge from controlled validation to clinical deployment, allowing experience to accumulate, unexpected behavior to be identified, and the same monitoring framework to continue after launch.
Link to the FDA discussion paper.”
Other posts featuring Joe Lennerz on OncoDaily.