File(s) under permanent embargo

Reason: Publisher requirement. Embargo date will be added following publication.

Quantifying ‘promising trials bias’ in randomized controlled trials in education

journal contribution
posted on 20.05.2022, 11:48 by Sam Sims, Jake Anders, Matthew InglisMatthew Inglis, Hugues Lortie-ForguesHugues Lortie-Forgues

Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect, if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results (ρ < α). We conduct a retrospective design analysis to quantify this bias across 22 such promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trials bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently scale up. Instead, the effect from the initial promising trial may simply be exaggerated.

History

School

  • Science

Department

  • Mathematics Education Centre

Published in

Journal of Research on Educational Effectiveness

Publisher

Taylor & Francis

Version

AM (Accepted Manuscript)

Publisher statement

This is an Accepted Manuscript of an article published by Taylor & Francis in Journal of Research on Educational Effectiveness on [date of publication], available at: http://www.tandfonline.com/[Article DOI].

Acceptance date

12/05/2022

ISSN

1934-5747

eISSN

1934-5739

Language

en

Depositor

Prof Matthew Inglis. Deposit date: 20 May 2022

Usage metrics

Categories

Exports