Author Correction: Comparing meta-analyses and preregistered multiple-laboratory replication projects

Correction to: Nature Human Behaviour https://doi.org/10.1038/10.1038/s41562-019-0787-z, published online 23 December 2019.

In the original version of this article, there was one code error and some data errors that required updating of Figs. 2–6, Table 1 and the Supplementary Information file. The core conclusions are unaffected, but some numbers and statements have been updated.

Fig. 2

Original and Corrected.

Fig. 3

Original and Corrected.

Fig. 4

Original and Corrected.

Fig. 5

Original and Corrected.

Fig. 6

Original and Corrected.

Original Table 1 Results for different indicators to assess the performance of the three meta-analysis bias-adjustment methods

Corrected Table 1 Results for different indicators to assess the performance of the three meta-analysis bias-adjustment methods

In Supplementary Tables 2–4, we have made the following corrections. The original effect size of the Sripada et al. (2014) study has been corrected from 0.29 to 0.68 (Supplementary Table 2), as the previous version incorrectly recorded the original estimate corresponding to a secondary replication estimate. The meta-analysis effect size corresponding to Rand et al. (2012) has been corrected from 0.056 to 0.117 (Supplementary Table 3) due to an error in converting the unstandardized effect size to Cohen’s d in the original manuscript. This meta-analysis did not report results in terms of Cohen’s d or a standard deviation of the outcome variable that could have been used to convert the result to Cohen’s d. We originally used an inappropriate formula to convert this result to Cohen’s d, and in the corrected version we instead converted the unstandardized effect size to Cohen’s d using the standard deviation from the replication study. Due to this same error in converting unstandardized effect sizes to Cohen’s d, the replication effect size corresponding to Srull & Wyer (1979) changed from 0.033 to 0.063 and the replication effect size for Schooler & Engstler-Schooler (1990) changed from 0.076 to 0.171 (Supplementary Table 4). The standard errors of the effect sizes for three replication studies, corresponding to Oppenheimer et al. (2009), Tversky & Kahneman (1981) and Husnu & Crisp (2010) were corrected and decreased as they were reconstructed from 99% confidence intervals that were erroneously assumed to be 95% confidence intervals (Supplementary Table 4). The standard error of the effect size for Monin & Miller (2001) has been corrected from 0.018 to 0.036 due to a reporting error (Supplementary Table 4).

The above corrections necessitated updates to Figs. 2–6 (see below for additional updates related to Figs. 5 and 6). For Fig. 2b there are still 12 statistically significant differences, but the meta-replication difference corresponding to Schooler & Engstler-Schooler (1990) is not statistically significant anymore, and the meta-replication difference corresponding to Rand et al. (2012) is now statistically significant (P = 0.0046) instead of the suggestive evidence (P = 0.047) reported in the original manuscript. In the manuscript text, the average unweighted meta-result has been corrected from 0.419 to 0.423, and the average unweighted replication result has been corrected from 0.155 to 0.163. The estimated random effects mean difference has been corrected from 0.263 (n = 15, z = 5.810, P < 0.001, 95% CI 0.175–0.352, 99.5% CI 0.136–0.391) to 0.260 (n = 15, z = 5.325, P < 0.001, 95% CI 0.164–0.355, 99.5% CI 0.123–0.396).

In Fig. 3 the point estimates changed somewhat compared to the original manuscript, as a result of the errors described above, but there are no substantive changes and the mean meta-replication effect size difference is still statistically significant in all the robustness tests and sub-group analyses.

For Fig. 4, the number of studies where the meta-effect size exceeds the original effect size changes from 4 to 3, and the number of studies where the original effect size exceeds the meta-analytic effect size changes from 10 to 11. As before, the difference is only statistically significant for Tversky & Kahneman (1981). In the manuscript text, the average unweighted original study effect size for the 14 studies in Fig. 4 has been corrected from 0.531 to 0.559 and the average unweighted meta-analytic effect size for the corresponding 14 meta-analyses has been corrected from 0.424 to 0.428. The random effects estimate of the difference between the original studies and the meta-analyses has been corrected from 0.09 to 0.112.

We have also made some corrections in the data and code used in the analyses with the bias-adjustment methods in Figs. 5 and 6 and Table 1. A standard error has been corrected due to a typo for one of the primary studies in the meta-analysis data for Graham, Haidt & Nosek (2009) (the standard error of the Graham et al. (2011) primary study for the USA), and an estimate that should not have been in that dataset has been removed (the overall meta-estimate was mistakenly included as one primary study in the initial analysis). The code for PET-PEESE has also been corrected, as the previous version incorrectly incorporated the between-study variance in the weights. Three 3PSM results, corresponding to Mazar et al. (2008), Schwarz, Strack & Mai (1991) and Schooler & Engstler-Schooler (1990), have also been corrected. As mentioned in the Methods section of the paper, we used the online app developed by Vevea & Coburn to estimate the 3PSM results, and in double-checking these results we found errors in the originally published 3PSM estimates for the three above mentioned studies.

For Fig. 5 and Table 1, the PET-PEESE results changed somewhat and the 3PSM results changed for the three studies where the results were replaced. For PET-PEESE the false negative rate decreased from 71.4% to 43% for the 0.5% level and from 75% to 50% for the 5% level, and the MDE decreased from 0.60 (0.46) for the 0.5% (5%) level to 0.45 (0.35). The overestimation factor for PET-PEESE increased from 0.95 to 1.31, and the root mean squared error increased from 0.22 to 0.26. For the 3PSM, the corrections resulted in one less false positive (Mazar et al., 2008) and one more false negative (Schooler & Engstler-Schooler, 1990), and the false positive rate at the 0.5% (5%) level decreased from 85.7% (100%) to 71.4% (83.3%) and the false negative rate at the 0.5% (5%) level increased from 14.3% (0%) to 28.6% (12.5%). The overestimation factor for 3PSM decreased from 2.49 to 2.24 and the root mean squared error decreased from 0.28 to 0.25.

In Fig. 6 and Table 1, the mean meta-replication difference is still statistically significant for the random effects model, 3PSM and Trim-and-Fill, and it is not statistically significant for PET-PEESE. The point estimate of the mean meta-replication difference changed from 0.265 to 0.261 for the random effects model, from 0.028 to 0.067 for PET-PEESE, from 0.235 to 0.210 for 3PSM and from 0.24 to 0.236 for Trim-and-Fill.

In the manuscript text, the mean Tau of the meta-analyses changed from 0.305 to 0.311, the Spearman correlation between Tau and the meta-replication difference changed from –0.305 (n = 15, P = 0.277) to –0.425 (n = 15, P = 0.114), and the Pearson correlation between Tau and the meta-replication difference changed from –0.406 (n = 15, P = 0.133) to –0.447 (n = 15, P = 0.095). The average effect size (standard error) reported in the 15 meta-analyses changed from 0.419 (0.051) to 0.423 (0.052), and the mean effect size (standard error) of the 15 meta-analyses using random effects analysis changed from 0.419 (0.055) to 0.423 (0.057).

We have also corrected a typo in the text where we write that three meta-analyses only have 80% power to detect a medium effect size; this has been corrected to one meta-analysis. Finally, in the updated version we include the references to all 15 replication studies and 15 meta-analyses included in the study in the reference list of the article (some of these references were previously only included in the reference list of the Supplementary Information file).