Studies to Interpret With Caution
Pereira TV, Horwitz RI, Ioannidis JP
JAMA. 2012;308:1676-1684
Pereira and colleagues sought to evaluate the frequency and features of very large effects in medical research. Using the Cochrane Database of Systematic Reviews, they examined binary outcomes with very large effects (odds ratio [OR], ≥ 5). Significant very large effects typically appeared in small trials with median number of events of 18 in first trials and 15 in subsequent trials. Of the very large effects observed in first and subsequently published trials, 90% and 98%, respectively, became smaller in meta-analyses that included other trials. The median OR declined from 11.88 to 4.20 for first trials and from 10.02 to 2.60 for subsequent trials. Pereira and colleagues concluded that most large treatment effects emerge from small studies, and when additional trials are performed, the effect sizes become much smaller. Well-validated large effects are uncommon.
Case reports and case series about wonderful new approaches to treating an illness are often followed by small studies that show much promise. However, large controlled trials do not necessarily confirm the initial impressions that generated all the excitement. Pereira and colleagues recommend that most large treatment effect estimates should be interpreted with caution because many are spurious findings or may represent substantial overestimations.
Abstract
Empirical Evaluation of Very Large Treatment Effects of Medical Interventions
Pereira TV, Horwitz RI, Ioannidis JP
JAMA. 2012;308:1676-1684
Large Effects and Small Trials
Pereira and colleagues sought to evaluate the frequency and features of very large effects in medical research. Using the Cochrane Database of Systematic Reviews, they examined binary outcomes with very large effects (odds ratio [OR], ≥ 5). Significant very large effects typically appeared in small trials with median number of events of 18 in first trials and 15 in subsequent trials. Of the very large effects observed in first and subsequently published trials, 90% and 98%, respectively, became smaller in meta-analyses that included other trials. The median OR declined from 11.88 to 4.20 for first trials and from 10.02 to 2.60 for subsequent trials. Pereira and colleagues concluded that most large treatment effects emerge from small studies, and when additional trials are performed, the effect sizes become much smaller. Well-validated large effects are uncommon.
Viewpoint
Case reports and case series about wonderful new approaches to treating an illness are often followed by small studies that show much promise. However, large controlled trials do not necessarily confirm the initial impressions that generated all the excitement. Pereira and colleagues recommend that most large treatment effect estimates should be interpreted with caution because many are spurious findings or may represent substantial overestimations.
Abstract
SHARE
