当前位置: X-MOL 学术Psychological Methods › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
Weighting schemes and incomplete data: A generalized Bayesian framework for chance-corrected interrater agreement.
Psychological Methods ( IF 10.929 ) Pub Date : 2021-11-12 , DOI: 10.1037/met0000412
Rutger van Oest 1 , Jeffrey M Girard 2
Affiliation  

Van Oest (2019) developed a framework to assess interrater agreement for nominal categories and complete data. We generalize this framework to all four situations of nominal or ordinal categories and complete or incomplete data. The mathematical solution yields a chance-corrected agreement coefficient that accommodates any weighting scheme for penalizing rater disagreements and any number of raters and categories. By incorporating Bayesian estimates of the category proportions, the generalized coefficient also captures situations in which raters classify only subsets of items; that is, incomplete data. Furthermore, this coefficient encompasses existing chance-corrected agreement coefficients: the S-coefficient, Scott’s pi, Fleiss’ kappa, and Van Oest’s uniform prior coefficient, all augmented with a weighting scheme and the option of incomplete data. We use simulation to compare these nested coefficients. The uniform prior coefficient tends to perform best, in particular, if one category has a much larger proportion than others. The gap with Scott’s pi and Fleiss’ kappa widens if the weighting scheme becomes more lenient to small disagreements and often if more item classifications are missing; missingness biases play a moderating role. The uniform prior coefficient often performs much better than the S-coefficient, but the S-coefficient sometimes performs best for small samples, missing data, and lenient weighting schemes. The generalized framework implies a new interpretation of chance-corrected weighted agreement coefficients: These coefficients estimate the probability that both raters in a pair assign an item to its correct category without guessing. Whereas Van Oest showed this interpretation for unweighted agreement, we generalize to weighted agreement.

中文翻译:

加权方案和不完整数据:机会校正的评估者间协议的广义贝叶斯框架。

Van Oest (2019) 开发了一个框架来评估评估者间对名义类别和完整数据的一致性。我们将此框架推广到名义或有序类别以及完整或不完整数据的所有四种情况。数学解决方案产生机会校正的一致性系数,该系数适用于惩罚评分者分歧的任何加权方案以及任何数量的评分者和类别。通过结合类别比例的贝叶斯估计,广义系数还捕获了评估者仅对项目子集进行分类的情况;也就是说,数据不完整。此外,该系数包含现有的机会校正一致性系数:S 系数、Scott 的 pi、Fleiss 的 kappa 和 Van Oest 的统一先验系数,所有这些都增加了加权方案和不完整数据的选项。我们使用模拟来比较这些嵌套系数。统一先验系数往往表现最好,特别是如果一个类别的比例比其他类别大得多。如果加权方案对小分歧更加宽松,并且如果缺少更多项目分类,则与 Scott 的 pi 和 Fleiss 的 kappa 的差距会扩大;缺失偏差起调节作用。统一先验系数通常比 S 系数表现更好,但 S 系数有时在小样本、缺失数据和宽松的加权方案中表现最佳。广义框架意味着对机会校正加权一致性系数的新解释:这些系数估计了一对评分者在没有猜测的情况下将一个项目分配到其正确类别的概率。鉴于 Van Oest 对未加权协议进行了这种解释,我们将其推广到加权协议。
更新日期:2021-11-12
down
wechat
bug