In this work, we propose a new generic multi-modality domain adaptation framework called Progressive Modality Cooperation (PMC) to transfer the knowledge learned from the source domain to the target domain by exploiting multiple modality clues ( e.g. , RGB and depth) under the multi-modality domain adaptation (MMDA) and the more general multi-modality domain adaptation using privileged information (MMDA-PI) settings. Under the MMDA setting, the samples in both domains have all the modalities. Through effective collaboration among multiple modalities, the two newly proposed modules in our PMC can select the reliable pseudo-labeled target samples, which captures the modality-specific information and modality-integrated information, respectively. Under the MMDA-PI setting, some modalities are missing in the target domain. Hence, to better exploit the multi-modality data in the source domain, we further propose the PMC with privileged information (PMC-PI) method by proposing a new multi-modality data generation (MMG) network. MMG generates the missing modalities in the target domain based on the source domain data by considering both domain distribution mismatch and semantics preservation, which are respectively achieved by using adversarial learning and conditioning on weighted pseudo semantic class labels. Extensive experiments on three image datasets and eight video datasets for various multi-modality cross-domain visual recognition tasks under both MMDA and MMDA-PI settings clearly demonstrate the effectiveness of our proposed PMC framework.

中文翻译：

---

用于多模态域自适应的渐进模态协作

在这项工作中，我们提出了一种新的通用多模态域适应框架，称为渐进模态合作（PMC），以通过利用多种模态线索将从源域学到的知识转移到目标域（ 例如 ，RGB和景深），以及使用特权信息（MMDA-PI）设置的更通用的多模态域适配。在MMDA设置下，两个域中的样本均具有所有模态。通过多种模式之间的有效协作，我们的PMC中两个新提议的模块可以选择可靠的伪标记目标样本，这些样本分别捕获特定于模式的信息和与模式集成的信息。在MMDA-PI设置下，目标域中缺少某些模式。因此，为了更好地利用源域中的多模态数据，我们通过提出一种新的多模态数据生成（MMG）网络，进一步提出了具有特权信息的PMC（PMC-PI）方法。MMG通过考虑域分布不匹配和语义保留来基于源域数据在目标域中生成丢失的模态，这分别是通过使用对抗性学习和对加权伪语义类标签进行条件化来分别实现的。在MMDA和MMDA-PI设置下，针对各种多模态跨域视觉识别任务的三个图像数据集和八个视频数据集的广泛实验清楚地证明了我们提出的PMC框架的有效性。