We scrutinize the argument that unsuccessful replications—and heterogeneous effect sizes more generally—may reflect an underappreciated influence of context characteristics. Notably, while some of these context characteristics may be conceptually irrelevant (as they merely affect psychometric properties of the measured/manipulated variables), others are conceptually relevant as they qualify a theory. Here, we present a conceptual and analytical framework that allows researchers to empirically estimate the extent to which effect size heterogeneity is due to conceptually relevant versus irrelevant context characteristics. According to this framework, contextual characteristics are conceptually relevant when the observed heterogeneity of effect sizes cannot be attributed to psychometric properties. As an illustrative example, we demonstrate that the observed heterogeneity of the “moral typecasting” effect, which had been included in the ManyLabs 2 replication project, is more likely attributable to conceptually relevant rather than irrelevant context characteristics, which suggests that the psychological theory behind this effect may need to be specified. In general, we argue that context dependency should be taken more seriously and treated more carefully by replication research.

我们仔细研究了以下论点，即不成功的复制——以及更普遍的异质效应大小——可能反映了背景特征的低估影响。值得注意的是，虽然这些背景特征中的一些可能在概念上无关（因为它们仅影响测量/操纵变量的心理测量特性），但其他特征在概念上是相关的，因为它们限定了理论。在这里，我们提出了一个概念和分析框架，允许研究人员凭经验估计效应大小异质性的程度是由于概念相关与不相关的上下文特征。根据这个框架，当观察到的效应量的异质性不能归因于心理测量特性时，上下文特征在概念上是相关的。作为一个说明性的例子，我们证明了观察到的“道德类型转换”效应的异质性，已包含在 ManyLabs 2 复制项目中，更有可能归因于概念相关而不是不相关的上下文特征，这表明这种效应背后的心理学理论可能需要被指定。总的来说，我们认为应该通过复制研究更认真地对待和更仔细地对待上下文依赖。