There is a frequent need to measure the degree of agreement among R observers who independently classify n subjects within K nominal or ordinal categories. The most popular methods are usually kappa‐type measurements. When R = 2, Cohen's kappa coefficient (weighted or not) is well known. When defined in the ordinal case while assuming quadratic weights, Cohen's kappa has the advantage of coinciding with the intraclass and concordance correlation coefficients. When R > 2, there are more discrepancies because the definition of the kappa coefficient depends on how the phrase ‘an agreement has occurred’ is interpreted. In this paper, Hubert's interpretation, that ‘an agreement occurs if and only if all raters agree on the categorization of an object’, is used, which leads to Hubert's (nominal) and Schuster and Smith's (ordinal) kappa coefficients. Formulae for the large‐sample variances for the estimators of all these coefficients are given, allowing the latter to illustrate the different ways of carrying out inference and, with the use of simulation, to select the optimal procedure. In addition, it is shown that Schuster and Smith's kappa coefficient coincides with the intraclass and concordance correlation coefficients if the first coefficient is also defined assuming quadratic weights.

中文翻译：

---

重新审视了休伯特（Hubert）的多评价卡帕（kappa）。

经常需要衡量R个观察者之间的一致程度，这些观察者将K个名义或有序类别中的n个主题独立分类。最受欢迎的方法通常是kappa型测量。当R  = 2时，科恩的kappa系数（加权或非加权）是众所周知的。当在假设二次权重的有序情况下定义时，科恩的kappa具有与类内和一致性相关系数一致的优势。当R  >  在图2中，存在更多差异，因为卡伯系数的定义取决于短语“已达成协议”的解释方式。在本文中，使用了休伯特的解释，即“仅当所有评估者都同意对对象的分类时，才会发生一致”，这导致了休伯特（名义）和舒斯特和史密斯（常态）的kappa系数。给出了所有这些系数的估计量的大样本方差的公式，从而使后者能够说明进行推断的不同方式，并通过仿真来选择最佳程序。此外，如果第一个系数也假设采用二次权重定义，则表明Schuster和Smith的kappa系数与类内和一致性相关系数一致。