Loughborough University
Browse
Quantifying Promising Trials Bias in Randomized Controlled Trials in Education.pdf (2.19 MB)

Quantifying “promising trials bias” in randomized controlled trials in education

Download (2.19 MB)
journal contribution
posted on 2023-10-20, 14:08 authored by Sam Sims, Jake Anders, Matthew InglisMatthew Inglis, Hugues Lortie-ForguesHugues Lortie-Forgues

Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect, if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results (ρ < α). We conduct a retrospective design analysis to quantify this bias across 22 such promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trials bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently scale up. Instead, the effect from the initial promising trial may simply be exaggerated.

History

School

  • Science

Department

  • Mathematics Education Centre

Published in

Journal of Research on Educational Effectiveness

Volume

16

Issue

4

Pages

663-680

Publisher

Taylor & Francis

Version

  • VoR (Version of Record)

Rights holder

© The Author(s)

Publisher statement

This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way.

Acceptance date

2022-05-06

Publication date

2022-07-11

Copyright date

2022

ISSN

1934-5747

eISSN

1934-5739

Language

  • en

Depositor

Prof Matthew Inglis. Deposit date: 20 May 2022

Usage metrics

    Loughborough Publications

    Categories

    No categories selected

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC