Annotating lesion locations by radiologists' manual marking is a key step to provide reference standard for the training and testing of a computer-aided detection system by supervised machine learning. Inter-reader variability is not uncommon in readings even by expert radiologists. This study evaluated the variability of the radiologist-identified pulmonary emboli (PEs) to demonstrate the importance of improving the reliability of the reference standard by a multi-step process for performance evaluation. In an initial reading of 40 CTPA PE cases, two experienced thoracic radiologists independently marked the PE locations. For markings from the two radiologists that did not agree, each radiologist re-read the cases independently to assess the discordant markings. Finally, for markings that still disagreed after the second reading, the two radiologists read together to reach a consensus. The variability of radiologists was evaluated by analyzing the agreement between two radiologists. For the 40 cases, 475 and 514 PEs were identified by radiologists R1 and R2 in the initial independent readings, respectively. For a total of 545 marks by the two radiologists, 81.5% (444/545) of the marks agreed but 101 marks in 36 cases differed. After consensus, 65 (64.4%) and 36 (35.6%) of the 101 marks were determined to be true PEs and false positives (FPs), respectively. Of these, 48 and 17 were false negatives (FNs) and 14 and 22 were FPs by R1 and R2, respectively. Our study demonstrated that there is substantial variability in reference standards provided by radiologists, which impacts the performance assessment of a lesion detection system. Combination of multiple radiologists' readings and consensus is needed to improve the reliability of a reference standard.

中文翻译：

---

放射科医生参考标准的变化以及在CT肺血管造影中检测肺栓塞的性能评估。

通过放射科医生的手动标记注释病变位置是关键步骤，可为通过监督机器学习训练和测试计算机辅助检测系统提供参考标准。阅读者之间的变异性在阅读中并不罕见，即使是放射线专家也是如此。这项研究评估了放射科医生确定的肺栓塞（PE）的变异性，以证明通过多步过程进行性能评估以提高参考标准的可靠性的重要性。在初步阅读40例CTPA PE病例中，两名经验丰富的胸腔放射科医生独立标记了PE部位。对于来自两位放射线医师的不同意的标记，每位放射线医师独立地重新阅读病例以评估不一致的标记。最后，对于在第二次阅读后仍然不一致的标记，两位放射科医生共同阅读以达成共识。通过分析两名放射科医生之间的协议来评估放射科医生的变异性。对于40例病例，放射科医生R1和R2在最初的独立读数中分别确定了475和514 PE。两位放射科医生总共对545个标记进行了同意，其中81.5％（444/545）的标记达成了共识，但36例中的101个标记存在差异。达成共识后，分别确定101个标记中的65个（64.4％）和36个（35.6％）是真实PE和假阳性（FP）。其中，R1和R2分别为48和17为假阴性（FN），14和22为FP。我们的研究表明，放射科医生提供的参考标准存在很大差异，这会影响病变检测系统的性能评估。多个放射科医生的结合