Zhi Ven Fong: Our study as June’s Feature article in Annals of Surgery

Zhi Ven Fong, Pancreas and Liver surgeon at Mayo Clinic, shared on X: 

“Grateful to Annals of Surgery for publishing our study as June’s Feature article!

Ever come across that stat sig (p<0.05), but with small diff btw study arms for it to be clinically meaningful? How pervasive is this?

We’ve got you covered in this thread: 

With recent advancements in the use of large data research and multicenter RCTs, it is important for us to distinguish clinical from statistical sig because even slightest diff in% can yield p<0.05 w large sample sizes.

Implications on trial design and drug approval are immense. Poster child below.

We tried to quantify how often papers would report stat sig without discerning clinical meaningfulness of between-group diff and if treatment change was recommended.

We queried 1 year’s worth of papers from Annals of Surgery, JAMA, Journal of Clinical Oncology, Journal of Surgical Research, and Journal of the American College of Surgeons (JACS) to get a good mix.

307 studies were reviewed. 54.7% had statistically significant outcomes. Only 8.5% discussed the clinical significance of between-group differences.

There were 7 (2.3%) studies where change in treatment was recommended, of which 5 did so without defining clinical significance.

As expected as well, for more studies reviewed, we correlated the reviewed studies’ sample sizes (x-axis) with the magnitude of between-group differences (y-axis) in their analyses and the association was quite linear.

Simply, it was a lot easier to hit p<0.05 with large sample sizes.

While this isn’t earth-shattering, intent was to reshine on an important issue.

This has implications on FDA/NICE approval of drugs and devices.

We join Dr Clavien’s call for action in his Editorial. We need to figure this out or someone else will for us!

This was a team effort by Harvard Medical School student, Ajami Gikandi, who’s applying for residency this year. Stud, rock star, keep an eye out for him!

And significant contributions from Julie Hallet, Bas Groot Koerkamp, Marc Besselink, Clancy J. Clark, Dana Gelb Safran, Raja Narayan, Harvey Mamon, and Joel Weissman.”

Source: Zhi Ven Fong/X