Increasingly, researchers are attempting to replicate published original studies by using large, multisite replication projects, at least 134 of which have been completed or are on going. These designs are promising to assess whether the original study is statistically consistent with the replications and to reassess the strength of evidence for the scientific effect of interest. However, existing analyses generally focus on single replications; when applied to multisite designs, they provide an incomplete view of aggregate evidence and can lead to misleading conclusions about replication success. We propose new statistical metrics representing firstly the probability that the original study's point estimate would be at least as extreme as it actually was, if in fact the original study were statistically consistent with the replications, and secondly the estimated proportion of population effects agreeing in direction with the original study. Generalized versions of the second metric enable consideration of only meaningfully strong population effects that agree in direction, or alternatively that disagree in direction, with the original study. These metrics apply when there are at least 10 replications (unless the heterogeneity estimate $\widehat{\tau }=0$, in which case the metrics apply regardless of the number of replications). The first metric assumes normal population effects but appears robust to violations in simulations; the second is distribution free. We provide R packages (Replicate and MetaUtility).

中文翻译：

---

多站点复制项目的新统计指标

研究人员越来越多地尝试通过使用大型的多站点复制项目来复制已发表的原始研究，其中至少有134个项目已经完成或正在进行中。这些设计有望评估原始研究在统计学上是否与重复实验一致，并重新评估感兴趣的科学效应的证据强度。但是，现有分析通常集中在单个复制上。当应用于多站点设计时，它们不能提供汇总证据的完整视图，并可能导致有关复制成功的误导性结论。我们提出了新的统计指标，首先代表原始研究的点估计至少与实际情况一样极端的可能性，如果实际上原始研究在统计学上与重复实验一致，其次，人口影响的估计比例与原始研究的方向一致。第二个度量的通用版本仅考虑有意义的强大人口效应，这些效应在方向上与原始研究一致，或者在方向上不同意。这些指标适用于至少有10次重复的情况（除非异质性估算$\widehat{\tau }=0$，在这种情况下，无论复制数量如何，指标均适用。第一个度量假设正常的人口效应，但是对于模拟中的违规表现出较强的鲁棒性。第二个是免费发行的。我们提供R程序包（复制和元实用程序）。