Phys.org May 31, 2023
A potential source of ambivalent empirical results on the same hypothesis is design heterogeneity—variation in true effect sizes across various reasonable experimental research protocols. An international team of researchers led by Austria invited independent research teams to contribute experimental designs to a crowd-sourced project. In a large-scale online data collection, 18,123 experimental participants were randomly allocated to 45 randomly selected experimental designs out of 95 submitted designs. They found a small adverse effect of competition on moral behavior in a meta-analysis of the pooled data. The crowd-sourced design of their study allowed for a clean identification and estimation of the variation in effect sizes above and beyond what could be expected due to sampling variance. They found substantial design heterogeneity indicating that the informativeness and generalizability of results based on a single experimental design are limited… read more. Open Access TECHNICAL ARTICLE
Relationship between effect sizes and experimental design quality… Credit: Proceedings of the National Academy of Sciences (2023), May 30, 2023, 120 (23) e2215572120