The evaluation of agreement among experts in a classification task is crucial in many situations (e.g., medical and psychological diagnosis, legal reports). Traditional indexes used to estimate interrater agreement (such as Cohen's j) simply count the number of observed agreements and correct them by removing chance agreements. In this article, we introduce a new theoretical framework for the evaluation of interrater agreement based on the possibility of adjusting the observed classifications conducted by the raters. This framework refers to the introduction and formalization of two concepts involved in the classification task: (a) the belonging measure of an object to a category and (b) the rater's belonging threshold, which is the minimally sufficient value of the belonging measure at which the rater will classify an object into a category. These factors are ignored by traditional indexes for interrater agreement, though their role may be decisive. Two Bayesian models are tested through a Monte Carlo simulation study to evaluate the accuracy of the new methodology for estimating raters' threshold and the actual degree of agreement between two independent raters. Results show that the computation of traditional indexes for interrater agreement on the adjusted classifications leads to a more accurate estimation of the experts' actual agreement. This improvement is greater when a large difference between raters' belonging thresholds is observed; when the difference is small, the proposed method provides similar results to those obtained in the simple observed classifications. Finally, an empirical application to the field of psychological assessment is presented to show how the method could be used in practice. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

中文翻译：

---

评估者阈值在评估评估者协议中的作用。

在许多情况下（例如，医学和心理诊断、法律报告），评估专家对分类任务的一致性至关重要。用于估计评价者间一致性的传统指标（例如 Cohen 的 j）只是简单地计算观察到的一致性的数量，并通过删除机会一致性来纠正它们。在本文中，我们介绍了一个新的理论框架，用于根据评估者对观察到的分类进行调整的可能性来评估评估者之间的一致性。该框架是指分类任务中涉及的两个概念的引入和形式化：（a）对象对类别的归属度量和（b）评估者的归属阈值，这是评估者将对象分类到类别中的归属度量的最小足够值。这些因素被传统的评价者协议指数忽略了，尽管它们的作用可能是决定性的。通过蒙特卡洛模拟研究测试了两个贝叶斯模型，以评估估计评估者阈值的新方法的准确性以及两个独立评估者之间的实际一致性程度。结果表明，计算调整后分类的评价者间一致性的传统指标可以更准确地估计专家的实际一致性。当观察到评分者的归属阈值之间存在较大差异时，这种改进会更大；当差异很小时，所提出的方法提供了与简单观察分类中获得的结果相似的结果。最后，提出了在心理评估领域的实证应用，以展示该方法如何在实践中使用。（PsycInfo 数据库记录 (c) 2021 APA，保留所有权利）。