Abstract

Multimodality datasets are becoming increasingly common in various domains to provide complementary information for predictive analytics. One significant challenge in fusing multimodality data is that the multiple modalities are not universally available for all samples due to cost and accessibility constraints. This situation results in a unique data structure called an Incomplete Multimodality Dataset. We propose a novel Incomplete-Multimodality Transfer Learning (IMTL) model that builds a predictive model for each sub-cohort of samples with the same missing modality pattern, and meanwhile couples the model estimation processes for different sub-cohorts to allow for transfer learning. We develop an Expectation-Maximization (EM) algorithm to estimate the parameters of IMTL and further extend it to a collaborative learning paradigm that is specifically valuable for patient privacy preservation in health care applications. We prove two advantageous properties of IMTL: the ability for out-of-sample prediction and a theoretical guarantee for a larger Fisher information compared with models without transfer learning. IMTL is applied to diagnosis and prognosis of Alzheimer’s disease at an early stage called Mild Cognitive Impairment using incomplete multimodality imaging data. IMTL achieves higher accuracy than competing methods without transfer learning.

摘要

多模态数据集在各个领域变得越来越普遍，为预测分析提供补充信息。融合多模态数据的一个重大挑战是，由于成本和可访问性的限制，多种模态并不普遍适用于所有样本。这种情况导致了一种独特的数据结构，称为不完整多模态数据集。我们提出了一种新颖的不完整多模态迁移学习（IMTL）模型，该模型为具有相同缺失模态模式的每个样本子群构建预测模型，同时耦合不同子群的模型估计过程以允许迁移学习。我们开发了一种期望最大化（EM）算法来估计 IMTL 的参数，并将其进一步扩展到协作学习范式，该范式对于医疗保健应用中的患者隐私保护特别有价值。我们证明了 IMTL 的两个有利特性：样本外预测的能力以及与没有迁移学习的模型相比对更大 Fisher 信息的理论保证。IMTL 使用不完整的多模态成像数据应用于阿尔茨海默病的早期诊断和预后（称为轻度认知障碍）。IMTL 比没有迁移学习的竞争方法实现了更高的准确性。与没有迁移学习的模型相比，样本外预测的能力以及更大的 Fisher 信息的理论保证。IMTL 使用不完整的多模态成像数据应用于阿尔茨海默病早期（称为轻度认知障碍）的诊断和预后。IMTL 比没有迁移学习的竞争方法实现了更高的准确性。与没有迁移学习的模型相比，样本外预测的能力以及更大的 Fisher 信息的理论保证。IMTL 使用不完整的多模态成像数据应用于阿尔茨海默病早期（称为轻度认知障碍）的诊断和预后。IMTL 比没有迁移学习的竞争方法实现了更高的准确性。