Intelligent reflecting surface (IRS) is envisioned to change the paradigm of wireless communications from “adapting to wireless channels” to “changing wireless channels”. However, current IRS configuration schemes, consisting of sub-channel estimation and passive beamforming in sequence, conform to the conventional model-based design philosophies and are difficult to be realized practically in the complex radio environment. To create the smart radio environment, we propose a model-free design of IRS control that is independent of the sub-channel channel state information (CSI) and requires the minimum interaction between IRS and the wireless communication system. We firstly model the control of IRS as a Markov decision process (MDP) and apply deep reinforcement learning (DRL) to perform real-time coarse phase control of IRS. Then, we apply extremum seeking control (ESC) as the fine phase control of IRS. Finally, by updating the frame structure, we integrate DRL and ESC in the model-free control of IRS to improve its adaptivity to different channel dynamics. Numerical results show the superiority of our proposed joint DRL and ESC scheme and verify its effectiveness in model-free IRS control without sub-channel CSI.

中文翻译：

---

使用深度强化学习的智能无线电智能反射面配置

智能反射面 (IRS) 旨在将无线通信的范式从“适应无线信道”转变为“改变无线信道”。然而，目前的IRS配置方案，依次由子信道估计和被动波束成形组成，符合传统的基于模型的设计理念，在复杂的无线电环境中难以实际实现。为了创建智能无线电环境，我们提出了一种独立于子信道信道状态信息 (CSI) 并且要求 IRS 与无线通信系统之间的交互最少的 IRS 控制的无模型设计。我们首先将 IRS 的控制建模为马尔可夫决策过程 (MDP)，并应用深度强化学习 (DRL) 来执行 IRS 的实时粗略相位控制。然后，我们应用极值搜索控制 (ESC) 作为 IRS 的精细相位控制。最后，通过更新帧结构，我们将 DRL 和 ESC 集成到 IRS 的无模型控制中，以提高其对不同通道动态的适应性。数值结果表明了我们提出的联合 DRL 和 ESC 方案的优越性，并验证了其在没有子通道 CSI 的无模型 IRS 控制中的有效性。