Multi-modal classification (MMC) aims to integrate the complementary information from different modalities to improve classification performance. Existing MMC methods can be grouped into two categories: traditional methods and deep learning-based methods. The traditional methods often implement fusion in a low-level original space. Besides, they mostly focus on the inter-modal fusion and neglect the intra-modal fusion. Thus, the representation capacity of fused features induced by them is insufficient. The deep learning-based methods implement the fusion in a high-level feature space where the associations among features are considered, while the whole process is implicit and the fused space lacks interpretability. Based on these observations, we propose a novel interpretative association-based fusion method for MMC, named AF. In AF, both the association information and the high-order information extracted from feature space are simultaneously encoded into a new feature space to help to train an MMC model in an explicit manner. Moreover, AF is a general fusion framework, and most existing MMC methods can be embedded into it to improve their performance. Finally, the effectiveness and the generality of AF are validated on 22 datasets, four typically traditional MMC methods adopting best modality, early, late and model fusion strategies and a deep learning-based MMC method.

中文翻译：

---

AF：一种基于关联的多模态分类融合方法


多模态分类（MMC）旨在整合来自不同模态的互补信息以提高分类性能。现有的MMC方法可以分为两类：传统方法和基于深度学习的方法。传统方法往往在低层原始空间中实现融合。此外，他们大多关注模态间融合而忽视模态内融合。因此，它们引起的融合特征的表示能力不足。基于深度学习的方法在高级特征空间中实现融合，其中考虑了特征之间的关联，但整个过程是隐式的，并且融合空间缺乏可解释性。基于这些观察，我们提出了一种新的基于解释关联的 MMC 融合方法，称为 AF。在AF中，从特征空间提取的关联信息和高阶信息同时编码到新的特征空间中，以帮助以显式方式训练MMC模型。而且，AF是一个通用的融合框架，大多数现有的MMC方法都可以嵌入其中以提高其性能。最后，在 22 个数据集、采用最佳模态的四种典型传统 MMC 方法、早期、晚期和模型融合策略以及基于深度学习的 MMC 方法中验证了 AF 的有效性和通用性。