19345747.2022.pdf (2.25 MB)
Download file

Quantifying “promising trials bias” in randomized controlled trials in education

Download (2.25 MB)
journal contribution
posted on 19.07.2022, 13:31 authored by Sam Sims, Jake Anders, Matthew InglisMatthew Inglis, Hugues Lortie-ForguesHugues Lortie-Forgues

Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect, if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results (ρ < α). We conduct a retrospective design analysis to quantify this bias across 22 such promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trials bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently scale up. Instead, the effect from the initial promising trial may simply be exaggerated.

History

School

  • Science

Department

  • Mathematics Education Centre

Published in

Journal of Research on Educational Effectiveness

Publisher

Taylor & Francis

Version

VoR (Version of Record)

Rights holder

© The Author(s)

Publisher statement

This is an Open Access Article. It is published by Taylor & Francis under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International Licence (CC BY-NC-ND). Full details of this licence are available at: https://creativecommons.org/licenses/by-nc-nd/4.0/

Acceptance date

06/05/2022

Publication date

2022-07-11

Copyright date

2022

ISSN

1934-5747

eISSN

1934-5739

Language

en

Depositor

Prof Matthew Inglis. Deposit date: 20 May 2022