Quantifying “promising trials bias” in randomized controlled trials in education
Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect, if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results (ρ < α). We conduct a retrospective design analysis to quantify this bias across 22 such promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trials bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently scale up. Instead, the effect from the initial promising trial may simply be exaggerated.
History
References
School
- Science
Department
- Mathematics Education Centre
Published in
Journal of Research on Educational EffectivenessPublisher
Taylor & FrancisVersion
- VoR (Version of Record)
Rights holder
© The Author(s)Publisher statement
This is an Open Access Article. It is published by Taylor & Francis under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International Licence (CC BY-NC-ND). Full details of this licence are available at: https://creativecommons.org/licenses/by-nc-nd/4.0/Acceptance date
2022-05-06Publication date
2022-07-11Copyright date
2022ISSN
1934-5747eISSN
1934-5739Publisher version
Language
- en