Agreement measures are useful tools to both compare different evaluations of the same diagnostic outcomes and validate new rating systems or devices. Cohen’s kappa (κ) certainly is the most popular agreement method between two raters, and proved its effectiveness in the last sixty years. In spite of that, this method suffers from some alleged issues, which have been highlighted since the 1970s; moreover, its value is strongly dependent on the prevalence of the disease in the considered sample. This work introduces a new agreement index, the informational agreement (IA), which seems to avoid some of Cohen’s kappa’s flaws, and separates the contribution of the prevalence from the nucleus of agreement. These goals are achieved by modelling the agreement—in both dichotomous and multivalue ordered-categorical cases—as the information shared between two raters through the virtual diagnostic channel connecting them: the more information exchanged between the raters, the higher their agreement. In order to test its fair behaviour and the effectiveness of the method, IA has been tested on some cases known to be problematic for κ, in the machine learning context and in a clinical scenario to compare ultrasound (US) and automated breast volume scanner (ABVS) in the setting of breast cancer imaging.

一致性测量是有用的工具，既可以比较对相同诊断结果的不同评估，又可以验证新的评级系统或设备。Cohen 的 kappa ( κ ) 无疑是两个评估者之间最流行的一致方法，并在过去 60 年中证明了它的有效性。尽管如此，这种方法仍存在一些所谓的问题，这些问题自 1970 年代以来就已被强调；此外，它的价值很大程度上取决于所考虑样本中疾病的流行程度。这项工作引入了一个新的协议索引，信息协议（IA)，这似乎避免了 Cohen 的 kappa 的一些缺陷，并将普遍性的贡献与协议核心区分开来。这些目标是通过对协议进行建模来实现的——在二分法和多值有序分类的情况下——作为两个评估者之间通过连接他们的虚拟诊断通道共享的信息：评估者之间交换的信息越多，他们的协议就越高。为了测试其公平行为和该方法的有效性，IA已经在一些已知对κ有问题的情况下进行了测试，在机器学习环境和临床场景中比较超声（US）和自动乳房体积扫描仪 （ABVS）在乳腺癌成像的设置。