We present a new approach to interpreting IRR that is empirical and contextualized. It is based upon benchmarking IRR against baseline measures in a replication, one of which is a novel cross-replication reliability (xRR) measure based on Cohen's kappa. We call this approach the xRR framework. We opensource a replication dataset of 4 million human judgements of facial expressions and analyze it with the proposed framework. We argue this framework can be used to measure the quality of crowdsourced datasets.

中文翻译：

---

交叉复制可靠性——解释评估者间可靠性的实证方法

我们提出了一种解释 IRR 的新方法，它是经验性的和情境化的。它基于针对复制中的基线测量对 IRR 进行基准测试，其中之一是基于 Cohen's kappa 的新型交叉复制可靠性 (xRR) 测量。我们称这种方法为 xRR 框架。我们开源了一个包含 400 万个人类面部表情判断的复制数据集，并使用所提出的框架对其进行分析。我们认为该框架可用于衡量众包数据集的质量。