In the era of sixth-generation (6G) wireless communications, integrated sensing and communications (ISAC) is recognized as a promising solution to upgrade the physical system by endowing wireless communications with sensing capability. Existing ISAC is mainly oriented to static scenarios with radio-frequency (RF) sensors being the primary participants, thus lacking a comprehensive environment feature characterization and facing a severe performance bottleneck in dynamic environments. To date, extensive surveys on ISAC have been conducted but are limited to summarizing RF-based radar sensing. Currently, some research efforts have been devoted to exploring multi-modal sensing-communication integration but still lack a comprehensive review. To fill the gap, we embark on an initial endeavor with the goal of establishing a unified framework of intelligent multi-modal sensing-communication integration by generalizing the concept of ISAC and providing a comprehensive review under this framework. Inspired by the human synesthesia, the so-termed Synesthesia of Machines (SoM) gives the clearest cognition of such an intelligent integration and details its paradigm for the first time. We commence by justifying the necessity and potential of the new paradigm. Subsequently, we offer a rigorous definition of SoM and zoom into the detailed paradigm, which is summarized as three operational modes realizing the integration. To facilitate SoM research, we overview the prerequisite of SoM research, that is, mixed multi-modal (MMM) datasets, and introduce our work. Built upon the MMM datasets, we introduce the mapping relationships between multi-modal sensing and communications, and discuss how channel modeling can be customized to support the exploration of such relationships. Afterward, aiming at giving a comprehensive survey on the current research status of multi-modal sensing-communication integration, we cover the technological review on SoM-enhance-based and SoM-concert-based applications in transceiver design and environment sensing. To corroborate the rationality and superiority of SoM, we also present simulation results related to dual-function waveform and predictive beamforming design tailored for dynamic scenarios. Finally, we propose some open issues and potential directions to inspire future research efforts on SoM.

中文翻译：

---

智能多模态传感通信集成：机器联觉

在第六代（6G）无线通信时代，集成传感与通信（ISAC）被认为是一种通过赋予无线通信传感能力来升级物理系统的有前途的解决方案。现有ISAC主要面向静态场景，以射频传感器为主要参与者，缺乏全面的环境特征表征，在动态环境下面临严重的性能瓶颈。迄今为止，已经对 ISAC 进行了广泛的调查，但仅限于总结基于射频的雷达传感。目前，一些研究工作致力于探索多模态传感通信集成，但仍缺乏全面的综述。为了填补这一空白，我们开始了初步的努力，目标是通过推广ISAC的概念并在此框架下进行全面审查，建立智能多模传感通信集成的统一框架。受人类联觉的启发，所谓的机器联觉（SoM）对这种智能集成给出了最清晰的认知，并首次详细介绍了其范式。我们首先证明新范式的必要性和潜力。随后，我们对SoM进行了严格的定义，并放大了详细的范式，将其概括为实现集成的三种操作模式。为了促进 SoM 研究，我们概述了 SoM 研究的先决条件，即混合多模态（MMM）数据集，并介绍了我们的工作。基于 MMM 数据集，我们介绍了多模态传感和通信之间的映射关系，并讨论了如何定制通道建模以支持对此类关系的探索。随后，为了全面了解多模态传感与通信集成的研究现状，我们对基于 SoM 增强和基于 SoM 协调的收发器设计和环境传感应用进行了技术综述。为了证实SoM的合理性和优越性，我们还提供了针对动态场景定制的双功能波形和预测波束形成设计的仿真结果。最后，我们提出了一些悬而未决的问题和潜在的方向，以激发未来 SoM 的研究工作。