We present a methodology motivated by a controlled trial designed to validate SPOT GRADE, a novel surgical bleeding severity scale. Briefly, the study was designed to quantify inter‐ and intra‐surgeon agreement for characterizing the severity of surgical bleeds via a Kappa statistic. Multiple surgeons were presented with a randomized sequence of controlled bleeding videos and asked to apply the rating system to characterize each wound. Each video was shown multiple times to quantify intra‐surgeon reliability, creating clustered data. In addition, videos within the same category may have had different classification probabilities due to changes in blood flow rates and wound sizes. In this work, we propose a new variance estimator for the Kappa statistic, for use in clustered data as well as heterogeneity among items within the same classification category. We then apply this methodology to data from the SPOT GRADE trial.

中文翻译：

---

在存在聚类数据和异构观察的情况下对Kappa统计量的方差估计。

我们介绍了一种旨在验证SPOT GRADE（一种新颖的手术出血严重程度评分表）的对照试验方法。简而言之，该研究旨在量化外科医生之间的共识，以通过Kappa统计量来表征手术出血的严重性。向多位外科医生提供了随机顺序的控制出血视频，并要求他们应用评分系统来表征每个伤口。每个视频都会显示多次，以量化手术中的可靠性，并创建聚类数据。另外，由于血流速度和伤口大小的变化，同一类别内的视频可能具有不同的分类概率。在这项工作中，我们为Kappa统计量提出了一个新的方差估算器，用于聚类数据以及同一分类类别中项目之间的异质性。然后，我们将此方法应用于SPOT GRADE试用版中的数据。