When haptic signals are integrated with existing audio-visual dominated multimedia applications, multi-modal services that can satisfy an individual's immersive experience have emerged. In order to support multi-modal services, cross-modal communications come into being. However, facing collaborative transmission and comprehensive processing requirements of audio, visual, and haptic signals as well as their influences for user's immersive experience, the research of cross-modal communications is still in its infancy and needs to tackle many technical challenges. Due to great successes and powerful abilities of artificial intelligence (AI), it can be expected to underpin cross-modal communications. In this article, we try to deal with issues in cross-modal communications by using AI technology. Specifically, we first adopt the federated learning paradigm to solve sparse data collection and privacy protection problems in the immersive experience description of multi-modal services. Then, we resort to the reinforcement learning paradigm to construct a joint optimization framework of caching, communication, and computation, realizing collaborative transmission of audio, visual, and haptic streams. Finally, we attempt to use the transfer learning paradigm to extract, transfer, and fuse knowledge, semantics, and characteristics from different modalities, recovering corrupted signals and promoting rendering effects at the receiver. Experimental results validate the effectiveness of the AI-enabled cross-modal communications strategies.

中文翻译：

---

人工智能支持的跨模式通信


当触觉信号与现有的视听主导的多媒体应用集成时，能够满足个人沉浸式体验的多模式服务就出现了。为了支持多模态服务，跨模态通信应运而生。然而，面对音频、视觉、触觉信号的协同传输和综合处理需求及其对用户沉浸式体验的影响，跨模态通信的研究仍处于起步阶段，需要解决许多技术挑战。由于人工智能（AI）的巨大成功和强大能力，它有望支撑跨模式通信。在这篇文章中，我们尝试利用人工智能技术来解决跨模态通信中的问题。具体来说，我们首先采用联邦学习范式来解决多模态服务的沉浸式体验描述中的稀疏数据收集和隐私保护问题。然后，我们利用强化学习范式构建缓存、通信和计算的联合优化框架，实现音频、视觉和触觉流的协同传输。最后，我们尝试使用迁移学习范式来提取、迁移和融合来自不同模态的知识、语义和特征，恢复损坏的信号并提升接收器的渲染效果。实验结果验证了人工智能支持的跨模式通信策略的有效性。