Conventional approaches to network control and resource allocation by allocating dedicated spectrum resources and separate infrastructure for massive Internet of Things (IoT) network are cost-inefficient. The concept of space-terrestrial integrated network (STIN), which is one of the enabling technologies and architectures for future 6 G wireless networks, can provide a solution to the network control and resource allocation problems for massive IoTs. In this paper, a novel STIN based network control and resource allocation problem is proposed for massive IoTs and solved through state of the art hierarchical deep actor-critic networks (H-DAC). The massive IoT networks spread over urban vicinity where cooperation can be possible. This is leveraged to negotiate a joint policy for per unit spectrum which the IoT networks are willing to pay. Deep actor-critic based reinforcement learning (RL) is used in this paper to solve the joint network control and resource allocation problem which is modeled as a utility maximization problem. The RL based algorithms solve the problem of cost per unit spectrum for the federated cloud of IoT networks and the data rate assigned to each IoT network and IoT devices. The algorithm also decides whether to transmit either through the space network or the terrestrial network. We validate performance of our proposed H-DAC scheme by comparing it with results of variants of the actor-critic based RL. We show that through proper system state formulation and reward design, the proposed H-DAC scheme outperforms the reference schemes with different network parameters and metrics.

中文翻译：

---

通过分层深度Actor-Critic强化学习实现天地一体化网络的联合网络控制和资源分配

通过为大规模物联网 (IoT) 网络分配专用频谱资源和单独的基础设施来进行网络控制和资源分配的传统方法成本效率低下。天地一体化网络（STIN）的概念是未来6G无线网络的使能技术和架构之一，可以为海量物联网的网络控制和资源分配问题提供解决方案。在本文中，针对大规模物联网提出了一种新的基于 STIN 的网络控制和资源分配问题，并通过最先进的分层深度演员-评论网络（H-DAC）解决了该问题。庞大的物联网网络遍布城市附近，在那里可以进行合作。这被用来协商物联网网络愿意支付的每单位频谱的联合政策。本文使用基于深度actor-critic 的强化学习（RL）来解决联合网络控制和资源分配问题，该问题被建模为效用最大化问题。基于 RL 的算法解决了物联网网络联合云的单位频谱成本以及分配给每个物联网网络和物联网设备的数据速率的问题。该算法还决定是通过空间网络还是通过地面网络进行传输。我们通过将其与基于 actor-critic 的 RL 变体的结果进行比较来验证我们提出的 H-DAC 方案的性能。我们表明，通过适当的系统状态制定和奖励设计，所提出的 H-DAC 方案优于具有不同网络参数和指标的参考方案。