Zhi Ven Fong: Our study is June’s Feature article in Annals of Surgery
Zhi Ven Fong, Pancreas and Liver surgeon at Mayo Clinic, shared a post on X:
“Grateful to Annals of Surgery for publishing our study as June’s Feature article!
Ever come across a paper that has statistical (p<0.05), but with small differnce between study arms for it to be clinically meaningful?
How pervasive is this? We’ve got you covered in this thread.
With recent increase in the use of large data research and multicenter RCTs, it is important for us to distinguish clinical from statistical sig because even slightest diff in% can yield p<0.05 with large sample sizes. Implications on trial design and drug approval are immense. Poster child:
We tried to quantify how often papers would report stat sig without discerning clinical meaningfulness of between-group diff and if treatment change was recommended.
307 studies were reviewed. 54.7% had statistically significant outcomes. Only 8.5% discussed the clinical significance of between-group differences. There were 7 (2.3%) studies where change in treatment was recommended, of which 5 did so without defining clinical significance.
How does this compare to the past? There was a study looking at the same issue & found that 19% of trials from 1975 to 1990 made an effort to determine if between-group differences were clinically meaningful. 30 years later, we’re not just no better, but fared worse at 8.5%.
As expected as well, for studies reviewed, we correlated the reviewed studies’ sample sizes (x-axis) with the magnitude of between-group differences (y-axis) in their analyses and the association was quite linear. Simply, it was a lot easier to hit p<0.05 with large sample sizes.
While this isn’t earth-shattering, intent was to reshine on an important issue. This has implications on FDA/NICE approval of drugs and devices. We join Dr Clavien’s call for action in his Editorial. We need to figure this out or someone else will for us!
This was an effort by Harvard Med student, Ajami Gikandi, who’s applying for residency this year. Stud, rock star, keep an eye out for him!
And significant contributions from Julie Hallet, Bas Groot Koerkam ,Marc Besselink, Clancy Clark, Dana Gelb Safran, Raja Narayan, Harvey Mamon and Joel Weissman.”
Nina Niu Sanford added, saying:
“Late to the party on this great paper by Zhi Ven Fong et al.
Only 8.5% of studies defined in methods what is clinically significant! Agree with authors that we over emphasize statistical sig/p-value while not focusing enough on whether something is even clinically meaningful!”
Source: Zhi Ven Fong/X and Nina Niu Sanford/X
Dr. Nina Niu Sanford is an Assistant Professor and Chief of Gastrointestinal Radiation Oncology at Harvard/Brigham and Women’s Hospital/Massachusetts General Hospital.
She specializes in treating gastrointestinal cancers and actively participates in clinical trials combining high-dose radiation therapy with immunotherapy. Additionally, she researches healthcare access disparities and conducts pan-cancer outcomes research using large databases.