Fusing data from multiple modalities provides more information to train machine learning systems. However, it is prohibitively expensive and time-consuming to label each modality with a large amount of data, which leads to a crucial problem of semi-supervised multi-modal learning. Existing methods suffer from either ineffective fusion across modalities or lack of theoretical guarantees under proper assumptions. In this paper, we propose a novel information-theoretic approach, namely \textbf{T}otal \textbf{C}orrelation \textbf{G}ain \textbf{M}aximization (TCGM), for semi-supervised multi-modal learning, which is endowed with promising properties: (i) it can utilize effectively the information across different modalities of unlabeled data points to facilitate training classifiers of each modality (ii) it has theoretical guarantee to identify Bayesian classifiers, i.e., the ground truth posteriors of all modalities. Specifically, by maximizing TC-induced loss (namely TC gain) over classifiers of all modalities, these classifiers can cooperatively discover the equivalent class of ground-truth classifiers; and identify the unique ones by leveraging limited percentage of labeled data. We apply our method to various tasks and achieve state-of-the-art results, including news classification, emotion recognition and disease prediction.

中文翻译：

---

TCGM：半监督多模态学习的信息理论框架

融合来自多种模态的数据为训练机器学习系统提供了更多信息。然而，用大量数据标记每个模态是非常昂贵和耗时的，这导致了半监督多模态学习的一个关键问题。现有的方法要么在模式之间进行无效融合，要么在适当的假设下缺乏理论保证。在本文中，我们提出了一种新的信息理论方法，即 \textbf{T}otal \textbf{C}orrelation \textbf{G}ain \textbf{M}aximization (TCGM)，用于半监督多模态学习，它被赋予了有前途的特性：(i) 它可以有效地利用未标记数据点的不同模态的信息来促进每种模态的分类器的训练 (ii) 它具有识别贝叶斯分类器的理论保证，即所有模态的真实后验。具体来说，通过在所有模态的分类器上最大化 TC 引起的损失（即 TC 增益），这些分类器可以协同发现真实分类器的等效类；并通过利用有限百分比的标记数据来识别独特的数据。我们将我们的方法应用于各种任务并获得最先进的结果，包括新闻分类、情感识别和疾病预测。这些分类器可以协同发现等价类的真值分类器；并通过利用有限百分比的标记数据来识别独特的数据。我们将我们的方法应用于各种任务并获得最先进的结果，包括新闻分类、情感识别和疾病预测。这些分类器可以协同发现等价类的真值分类器；并通过利用有限百分比的标记数据来识别独特的数据。我们将我们的方法应用于各种任务并获得最先进的结果，包括新闻分类、情感识别和疾病预测。