Clinical decision-making in oncology involves multimodal data such as radiology scans, molecular profiling, histopathology slides, and clinical factors. Despite the importance of these modalities individually, no deep learning framework to date has combined them all to predict patient prognosis. Here, we predict the overall survival (OS) of glioma patients from diverse multimodal data with a Deep Orthogonal Fusion (DOF) model. The model learns to combine information from multiparametric MRI exams, biopsy-based modalities (such as H&E slide images and/or DNA sequencing), and clinical variables into a comprehensive multimodal risk score. Prognostic embeddings from each modality are learned and combined via attention-gated tensor fusion. To maximize the information gleaned from each modality, we introduce a multimodal orthogonalization (MMO) loss term that increases model performance by incentivizing constituent embeddings to be more complementary. DOF predicts OS in glioma patients with a median C-index of 0.788 +/- 0.067, significantly outperforming (p=0.023) the best performing unimodal model with a median C-index of 0.718 +/- 0.064. The prognostic model significantly stratifies glioma patients by OS within clinical subsets, adding further granularity to prognostic clinical grading and molecular subtyping.

中文翻译：

---

深度正交融合：整合放射学、病理学、基因组和临床数据的多模式预后生物标志物发现

肿瘤学的临床决策涉及多模式数据，例如放射学扫描、分子分析、组织病理学幻灯片和临床因素。尽管这些方式单独具有重要性，但迄今为止还没有深度学习框架将它们全部结合起来预测患者的预后。在这里，我们使用深度正交融合 (DOF) 模型从不同的多模态数据中预测神经胶质瘤患者的总生存期 (OS)。该模型学习将来自多参数 MRI 检查、基于活检的模式（例如 H&E 幻灯片图像和/或 DNA 测序）和临床变量的信息组合成综合的多模式风险评分。通过注意力门控张量融合学习和组合来自每种模态的预后嵌入。为了最大限度地利用从每种模式中收集到的信息，我们引入了一个多模态正交化 (MMO) 损失项，它通过激励成分嵌入更加互补来提高模型性能。DOF 预测神经胶质瘤患者的 OS，C 指数中位数为 0.788 +/- 0.067，显着优于（p = 0.023）表现最佳的单峰模型，C 指数中位数为 0.718 +/- 0.064。预后模型通过临床亚组内的 OS 对神经胶质瘤患者进行了显着分层，为预后临床分级和分子亚型分型增加了进一步的粒度。