ASCO24 Updates: Dr. Tony Hung on GPT-4’s Role in Clinical Trial Recommendations for Head and Neck Cancer | OncoDaily

The American Society of Clinical Oncology (ASCO) Annual Meeting is one of the largest and most prestigious conferences in the field of oncology. This year, the meeting took place from May 31 to June 4 in Chicago, Illinois. The event gathers oncologists, researchers, and healthcare professionals from around the world to discuss the latest advancements in cancer research, treatment, and patient care. Keynote sessions, research presentations, and panel discussions are typically part of the agenda, providing attendees with valuable insights into emerging trends and innovations in oncology.

This year, OncoDaily was at ASCO 2024 for the first time covering the meeting on-site. We had the pleasure of interviewing researchers who summarized the highlights of their work.

In this video, Tony Hung from Memorial Sloan Kettering Cancer Center, shares their insights on ‘Performance of a trained large language model to provide clinical trial recommendation in a head and neck cancer population.’

Hi, my name is Tony Hung. I am a head and neck medical oncologist at the Memorial Sloan Kettering Cancer Center. So my abstract really summarized the result of a study that we conducted in evaluating the performance of a trained large language model powered by GPT-4 by OpenAI in providing a clinical trial recommendation for a head and neck patient population.

Specifically, we hypothesized that this trained large language model would provide clinical trial recommendation that are comparable to medical oncologists at MSK, where the study is being conducted. So prior to our study, we do know that an untrained large language model like GPT-4 can have the ability to answer a medical oncology examination question like the USMLE. However, these model have yet really demonstrated suitable performance in the clinical setting and in practice.

So what we did was we trained GPT-4 using a dataset provided by a clinical trial knowledge management application that I have developed called Lookup Trial. And we build a chatbot interface and integrate the model into the interface to provide clinical trial recommendation. And after which we then collected consecutive new patient cases in the head and neck oncology department.

And we categorize these patient cases by the patient’s cancer diagnosis, cancer stage, treatment setting, as well as the respective clinical trial recommendation made by the medical oncologist. We then prompted the chatbot to give us a clinical trial recommendation asking, given a patient with a specific diagnosis, stage, treatment setting, what are the possible clinical trial at MSK?

And we then compare these responses, the GPT-4 response, with that of the actual clinician recommendation. We measure the performance specifically by the respond precision, also known as the positive predictive value, the respond recall, which are the sensitivity, and as well as the F1 score, which are the harmonic mean of the precision and recall.

And what we found was quite impressive in that over the two-month period where we collected patient case from November to December of 2023, we analyzed about 178 cases. At the time, there were about 56 clinical trial that are ongoing in the head and neck oncology service. And GPT-4 really demonstrate what I would consider a moderate performance, achieving a precision of 63%, a recall of 100% with a harmonic mean of 0.77. Comparatively, this performance really exceed the historical performance of untrained GPT in providing oncology recommendation by about four to 20-folds.

So what this pilot or proof-of-concept study suggests is the potential of these trained large-language model to a clinical trial search, but also the possibility of these model in supporting oncology provider in the clinical trial accrual process. There’s definitely a lot more work to be done, and including really how we can optimize the performance of these large-language model, as well as implementing these model in the real-world setting to enhance, for example, clinical trial participation.

More videos and content from ASCO 2024 on OncoDaily.