Opinion

Mar 11, 2025, 11:45

Douglas Flora: Should AI-driven clinical decision support be regulated?

Recently, npj Digital Medicine published an article discussing how unregulated large language models produce medical device-like output (authored by Gary Weissman, Toni Mankowitz, and Genevieve Kanter).

While technology advancements like this are revolutionizing medicine and healthcare, there are many unresolved questions on the safe and efficient implementation of these methods in clinical practice.

Dr. Douglas Flora, Executive Medical Director of Oncology Services at St. Elizabeth Healthcare and President-Elect of the Association of Cancer Care Centers, Editor-in-Chief of AI in Precision Oncology, recently commented on the above-mentioned paper in his latest LinkedIn post:

“AI in Healthcare: A Regulatory Wake-Up Call?

Large language models (LLMs) like GPT-4 and Llama-3 are showing incredible promise in clinical decision support.

But here’s the catch: they’re not regulated as medical devices, and yet, they’re already generating recommendations that look a lot like regulated medical guidance.

A recent study found that even when prompted to avoid device-like recommendations, these AI models often provided clinical decision support in ways that could MEET FDA CRITERIA for a medical device.

In some cases, their responses aligned with established medical standards – but in others, they ventured into HIGH-RISK TERRITORY, making treatment recommendations that should only come from trained professionals.

This raises a big question: Should AI-driven clinical decision support be regulated? And if so, how do we balance innovation with patient safety? Right now, there’s no clear framework for LLMs used by non-clinicians in critical situations.

What does this mean for healthcare professionals?

AI is advancing fast, and while it can be a powerful tool, it’s crucial to recognize its limitations.

For regulators?

There’s an urgent need to define new oversight models that account for generative AI’s unique capabilities.

For AI developers?

Transparency, accuracy, and adherence to safety standards will be key to building trust in medical AI applications.

As AI continues to evolve, we’re entering uncharted territory. The conversation about regulation isn’t just theoretical – it’s becoming a necessity.

What do you think? Should AI in clinical decision support be regulated like a medical device? Let’s discuss.” – Douglas Flora.

Read the article below and let us know what you think. Find out more about AI in Healthcare from Dr. Flora on OncoDaily.

“Unregulated large language models produce medical device-like output”

Authors: Gary Weissman, Toni Mankowitz, and Genevieve Kanter.

Douglas Flora

ACCC AI in healthcare AI in medicine AI in Precision Oncology artificial intelligence Association of Cancer Care Centers cancer Clinical Decision Support Douglas Flora Gary Weissman Genevieve Kanter GPT-4 Large Language Models Llama-3 LLMs npj Digital Medicine OncoDaily Oncology St. Elizabeth Healthcare Toni Mankowitz Unregulated AI

VERITAC-2 Trial Update: Vepdegestrant Meets Primary Endpoint in ESR1-Mutant Breast Cancer

New Paper Alert: HARMONi-2 study of Ivonescimab vs pembrolizumab for PD-L1-positive non-small cell lung cancer

We are thrilled to welcome Dr. Jasmine Kamboj to the OncoDaily Editorial Board!

Oncologist Near Me – Fedro Alessandro Peccatori: Leading Expert in Gynecological Oncology

MATTERHORN trial - FLOT plus durvalumab will become the new standard of care in the preoperative setting for gastroesophageal cancer