Ravi Parikh: Performance Drift in a National Population Health Risk Algorithm
Ravi Parikh, Oncologist and Director of the Human-Algorithm Collaboration Lab (HACLab) at Winship Cancer Institute of Emory University, shared a post on LinkedIn about a paper he co-authored with colleagues published in JAMA Health Forum:
“New paper alert! Our work on performance drift, or post-deployment changes in algorithm performance, in a widely-used U.S. Department of Veterans Affairs algorithm is now published in JAMA Health Forum, with an accompanying commentary by Andrew Wong and Jeremy Sussman.
We study the VA CAN algorithm, which uses patient demographic and health utilization data to generate a score that measures the likelihood of a 90-day combined hospitalization and/or mortality outcome among VA primary care patients. The score can be used to inform clinical decisions and VA quality metrics.
We find that between 2016 and 2021, the CAN algorithm decreased in performance, with a 4% drop in its positive predictive value (precision) (i.e. PPV) and smaller but clinically meaningful shifts in true positive rate (recall) and false positive rate. These findings indicate a greater inaccuracy in classifying Veterans’ by their risk for the combined outcome.
Mechanisms of drift include shifts in covariate and/or outcome data distributions in real-world settings. We find evidence of both:
- shifts in demographics, utilization, and lab values – especially during COVID-19; and
- a decline in the outcome rate itself.
Drift can undermine the reliability of quality metrics built on algorithm classifications. For example, a metric tracking palliative care visits among algorithm-classified ‘high-risk’ veterans was increasingly dominated by false positives – rising from ~82% to ~86% between 2018–2020.
As the use of clinical algorithms grows, so does the need for robust monitoring and regulation of these tools. Our work supports national calls for better postmarket algorithm audits (similar to FDA surveillance for drugs), robust monitoring frameworks, and institutional capacity to detect and mitigate drift.
Work was led by Likhitha Kolla, MD/PhD student, Human Algorithm Collaboration Lab and Penn Medicine Department of Biostatistics, Epidemiology and Informatics, co-mentored by Jinbo Chen and myself.”
Title: Performance Drift in a Nationally Deployed Population Health Risk Algorithm in the US Veterans Health Administration
Authors: Likhitha Kolla, Kristin Linn, Amol S. Navathe, Craig Kreisler, Christopher B. Roberts, Sae-Hwan Park, Harvineet Singh, Jean Feng, Jinbo Chen, Ravi B. Parikh
You can read the Full Article on JAMA Health Forum.
More posts featuring Ravi Parikh.
-
Challenging the Status Quo in Colorectal Cancer 2024
December 6-8, 2024
-
ESMO 2024 Congress
September 13-17, 2024
-
ASCO Annual Meeting
May 30 - June 4, 2024
-
Yvonne Award 2024
May 31, 2024
-
OncoThon 2024, Online
Feb. 15, 2024
-
Global Summit on War & Cancer 2023, Online
Dec. 14-16, 2023