Objectives

To compare different Machine Learning (ML) Natural Language Processing (NLP) methods to classify radiology reports in orthopaedic trauma for the presence of injuries. Assessing NLP performance is a prerequisite for downstream tasks and therefore of importance from a clinical perspective (avoiding missed injuries, quality check, insight in diagnostic yield) as well as from a research perspective (identification of patient cohorts, annotation of radiographs).

Methods

Datasets of Dutch radiology reports of injured extremities (n = 2469, 33% fractures) and chest radiographs (n = 799, 20% pneumothorax) were collected in two different hospitals and labeled by radiologists and trauma surgeons for the presence or absence of injuries. NLP classification was applied and optimized by testing different preprocessing steps and different classifiers (Rule-based, ML, and Bidirectional Encoder Representations from Transformers (BERT)). Performance was assessed by F1-score, AUC, sensitivity, specificity and accuracy.

Results

The deep learning based BERT model outperforms all other classification methods which were assessed. The model achieved an F1-score of (95 ± 2)% and accuracy of (96 ± 1)% on a dataset of simple reports (n= 2469), and an F1 of (83 ± 7)% with accuracy (93 ± 2)% on a dataset of complex reports (n= 799).

Conclusion

BERT NLP outperforms traditional ML and rule-base classifiers when applied to Dutch radiology reports in orthopaedic trauma.

中文翻译：

---

基于机器学习的骨科创伤放射学报告的自然语言处理

目标

比较不同的机器学习 (ML) 自然语言处理 (NLP) 方法，以对骨科创伤中的放射学报告是否存在损伤进行分类。评估 NLP 性能是下游任务的先决条件，因此从临床角度（避免漏诊、质量检查、诊断率洞察）和研究角度（患者队列的识别、X 光片注释）都很重要。

方法

荷兰放射学报告的四肢受伤（n  = 2469，33% 骨折）和胸片（n  = 799，20% 气胸）的数据集在两家不同的医院收集，并由放射科医生和创伤外科医生标记是否存在损伤。通过测试不同的预处理步骤和不同的分类器（基于规则、ML 和来自 Transformers (BERT) 的双向编码器表示）来应用和优化 NLP 分类。性能通过 F1 分数、AUC、敏感性、特异性和准确性进行评估。

结果

基于深度学习的 BERT 模型优于评估的所有其他分类方法。该模型在简单报告数据集 (n= 2469) 上实现了 (95 ± 2)% 的 F1 分数和 (96 ± 1)% 的准确度，以及在准确度 (93 ± 2)% 在复杂报告的数据集上 (n=799)。

结论

当应用于荷兰骨科创伤放射学报告时，BERT NLP 优于传统的机器学习和规则库分类器。