Empirical evaluations of replication have become increasingly common, but there has been no unified approach to doing so. Some evaluations conduct only a single replication study while others run several, usually across multiple laboratories. Designing such programs has largely contended with difficult issues about which experimental components are necessary for a set of studies to be considered replications. However, another important consideration is that replication studies be designed to support sufficiently sensitive analyses. For instance, if hypothesis tests are to be conducted about replication, studies should be designed to ensure these tests are well-powered; if not, it can be difficult to determine conclusively if replication attempts succeeded or failed. This paper describes methods for designing ensembles of replication studies to ensure that they are both adequately sensitive and cost-efficient. It describes two potential analyses of replication studies—hypothesis tests and variance component estimation—and approaches to obtaining optimal designs for them. Using these results, it assesses the statistical power, precision of point estimators and optimality of the design used by the Many Labs Project and finds that while it may have been sufficiently powered to detect some larger differences between studies, other designs would have been less costly and/or produced more precise estimates or higher-powered hypothesis tests.

中文翻译：

---

重复研究的设计

对复制的实证评估变得越来越普遍，但还没有统一的方法来这样做。一些评估只进行一次重复研究，而另一些则进行多次重复研究，通常跨越多个实验室。设计这样的程序在很大程度上要解决一些难题，即哪些实验组件对于一组被认为是重复的研究是必要的。然而，另一个重要的考虑是复制研究的设计是为了支持足够敏感的分析。例如，如果要进行关于复制的假设检验，则应设计研究以确保这些检验具有良好的效力；如果不是，则很难确定复制尝试是成功还是失败。本文描述了设计复制研究集合的方法，以确保它们具有足够的敏感性和成本效益。它描述了重复研究的两种潜在分析——假设检验和方差分量估计——以及为它们获得最佳设计的方法。使用这些结果，它评估了 Many Labs Project 使用的设计的统计功效、点估计器的精度和最优性，并发现虽然它可能有足够的功效来检测研究之间的一些更大的差异，但其他设计的成本会更低和/或产生更精确的估计或更高功率的假设检验。它描述了重复研究的两种潜在分析——假设检验和方差分量估计——以及为它们获得最佳设计的方法。使用这些结果，它评估了 Many Labs Project 使用的设计的统计功效、点估计器的精度和最优性，并发现虽然它可能有足够的功效来检测研究之间的一些更大的差异，但其他设计的成本会更低和/或产生更精确的估计或更高功率的假设检验。它描述了重复研究的两种潜在分析——假设检验和方差分量估计——以及为它们获得最佳设计的方法。使用这些结果，它评估了 Many Labs Project 使用的设计的统计功效、点估计器的精度和最优性，并发现虽然它可能有足够的功效来检测研究之间的一些更大的差异，但其他设计的成本会更低和/或产生更精确的估计或更高功率的假设检验。