Resource allocation and spectrum management are two major challenges in the massive scale deployment of Internet of Things (IoT) and Machine-to-Machine (M2M) communication. Furthermore, the large number of devices per unit area in IoT networks also leads to congestion, network overload, and deterioration of the Signal to Noise Ratio (SNR). To address these problems, efficient resource allocation play a pivotal role in optimizing the throughput, delay, and power management of IoT networks. To this end, most of the existing resource allocation mechanisms are centralized and do not gracefully support the heterogeneous and dynamic IoT networks. Therefore, distributed and Machine Learning (ML)-based approaches are essential. However, distributed resource allocation techniques also have scalability problem with large number of devices whereas the ML-based approaches are currently scarce in the literature. In this paper, we propose a new distributed block-based Q-learning algorithm for slot scheduling in the smart devices and Machine Type Communication Devices (MTCDs) participating in clustered IoT networks. We furthermore, propose various reward schemes for the evolution of Q-values in the proposed scheme and, discuss and evaluate their effect on the distributed model. Our goal is to avoid inter- and intra-cluster interference, and to improve the Signal to Interference Ratio (SIR) by employing frequency diversity in a multi-channel system. Through extensive simulations, we analyze the effects of the distributed slot-assignment (with respect to varying SIR) on the convergence rate and the convergence probability. Our theoretical analysis and simulations validate the effectiveness of our proposed method where, (i) a suitable slot with acceptable SIR levels is allocated to each MTCD, and (ii) IoT network can efficiently converge to a collision-free transmission causing minimum intra-cluster interference. The network convergence is achieved through each MTCD's learning ability during the distributed slot allocation.

中文翻译：

---

一种新的基于块的强化学习方法，用于集群物联网网络中的分布式资源分配

资源分配和频谱管理是物联网 (IoT) 和机器对机器 (M2M) 通信大规模部署的两大挑战。此外，物联网网络中每单位面积的大量设备也会导致拥塞、网络过载和信噪比（SNR）恶化。为了解决这些问题，有效的资源分配在优化物联网网络的吞吐量、延迟和电源管理方面发挥着关键作用。为此，现有的大部分资源分配机制都是集中式的，不能很好地支持异构和动态的物联网网络。因此，分布式和基于机器学习 (ML) 的方法必不可少。然而，分布式资源分配技术还存在大量设备的可扩展性问题，而基于 ML 的方法目前在文献中很少见。在本文中，我们提出了一种新的基于分布式块的 Q 学习算法，用于参与集群物联网网络的智能设备和机器类型通信设备 (MTCD) 中的时隙调度。此外，我们还为所提出方案中 Q 值的演变提出了各种奖励方案，并讨论和评估了它们对分布式模型的影响。我们的目标是避免集群间和集群内干扰，并通过在多信道系统中采用频率分集来提高信号干扰比 (SIR)。通过大量的模拟，我们分析了分布式时隙分配（相对于变化的 SIR）对收敛速度和收敛概率的影响。我们的理论分析和模拟验证了我们提出的方法的有效性，其中，（i）为每个 MTCD 分配一个具有可接受 SIR 水平的合适时隙，以及（ii）物联网网络可以有效地收敛到无冲突传输，从而导致集群内最小化干涉。网络收敛是通过每个 MTCD 在分布式时隙分配过程中的学习能力来实现的。(ii) 物联网网络可以有效地收敛到无冲突传输，从而将集群内干扰降至最低。网络收敛是通过每个 MTCD 在分布式时隙分配过程中的学习能力来实现的。(ii) 物联网网络可以有效地收敛到无冲突传输，从而将集群内干扰降至最低。网络收敛是通过每个 MTCD 在分布式时隙分配过程中的学习能力来实现的。