Magnetic Resonance Imaging (MRI) is a pivotal neuroimaging technique capable of generating images with various contrasts, known as multi-modal images. The integration of these diverse modalities is essential for improving model performance across various tasks. However, in real clinical scenarios, acquiring MR images for all modalities is frequently hindered by factors such as patient comfort and scanning costs. Therefore, effectively fusing different modalities to synthesize missing modalities has become a research hot-spot in the field of smart healthcare, particularly in the context of the Internet of Medical Things (IoMT). In this study, we introduce a multi-modal coordinated fusion network (MCF-Net) with Patch Complementarity Pre-training. This network leverages the complementarity and correlation between different modalities to make the fusion of multi-modal MR images, addressing challenges in the IoMT. Specifically, we first employ a Patch Complementarity Mask Autoencoder (PC-MAE) for self-supervised pre-training. The complementarity learning mechanism is introduced to align masks and visual annotations between two modalities. Subsequently, a dual-branch MAE architecture and a shared encoder–decoder are adopted to facilitate cross-modal interactions within mask tokens. Furthermore, during the fine-tuning phase, we incorporate an Attention-Driven Fusion (ADF) module into the MCF-Net. This module synthesizes missing modal images by fusion of multi-modal features from the pre-trained PC-MAE encoder. Additionally, we leverage the pre-trained encoder to extract high-level features from both synthetic and corresponding real images, ensuring consistency throughout the training process. Our experimental findings showcase a notable enhancement in performance across various modalities with our fusion method, outperforming state-of-the-art techniques.

中文翻译：

---

多模态 MRI 融合与补丁互补预训练，用于基于医疗物联网的智能医疗

磁共振成像 (MRI) 是一种关键的神经成像技术，能够生成具有各种对比度的图像，称为多模态图像。这些不同模式的集成对于提高各种任务的模型性能至关重要。然而，在实际临床场景中，获取所有模式的 MR 图像经常受到患者舒适度和扫描成本等因素的阻碍。因此，有效融合不同模态以合成缺失模态已成为智能医疗领域的研究热点，特别是在医疗物联网（IoMT）背景下。在本研究中，我们引入了具有补丁互补预训练的多模态协调融合网络（MCF-Net）。该网络利用不同模态之间的互补性和相关性来融合多模态 MR 图像，解决 IoMT 中的挑战。具体来说，我们首先采用补丁互补掩模自动编码器（PC-MAE）进行自监督预训练。引入互补学习机制来对齐两种模式之间的掩模和视觉注释。随后，采用双分支 MAE 架构和共享编码器-解码器来促进掩码令牌内的跨模式交互。此外，在微调阶段，我们将注意力驱动融合（ADF）模块合并到 MCF-Net 中。该模块通过融合预训练 PC-MAE 编码器的多模态特征来合成缺失的模态图像。此外，我们利用预训练的编码器从合成图像和相应的真实图像中提取高级特征，确保整个训练过程的一致性。我们的实验结果表明，我们的融合方法在各种模式下的性能显着提高，优于最先进的技术。