Multimodal machine learning (MML) aims to understand the world from multiple related modalities. It has attracted much attention as multimodal data has become increasingly available in real-world application. It is shown that MML can perform better than single-modal machine learning, since multi-modalities containing more information which could complement each other. However, it is a key challenge to fuse the multi-modalities in MML. Different from previous work, we further consider the side-information, which reflects the situation and influences the fusion of multi-modalities. We recover multimodal label distribution (MLD) by leveraging the side-information, representing the degree to which each modality contributes to describing the instance. Accordingly, a novel framework named multimodal label distribution learning (MLDL) is proposed to recover the MLD, and fuse the multimodalities with its guidance to learn an in-depth understanding of the jointly feature representation. Moreover, two versions of MLDL are proposed to deal with the sequential data. Experiments on multimodal sentiment analysis and disease prediction show that the proposed approaches perform favorably against state-of-the-art methods.

多模态机器学习 (MML) 旨在从多种相关模态理解世界。随着多模态数据在实际应用中变得越来越可用，它引起了很多关注。结果表明，MML 的性能优于单模态机器学习，因为多模态包含更多可以相互补充的信息。然而，融合 MML 中的多模态是一个关键的挑战。与之前的工作不同，我们进一步考虑了边信息，它反映了情况并影响了多模态的融合。我们通过利用边信息来恢复多模态标签分布（MLD），代表每个模态对描述实例的贡献程度。因此，提出了一种名为多模态标签分布学习 (MLDL) 的新框架来恢复 MLD，并将多模态与其指导融合，以深入了解联合特征表示。此外，提出了两个版本的 MLDL 来处理顺序数据。多模态情感分析和疾病预测的实验表明，所提出的方法与最先进的方法相比表现良好。