This paper proposes a learning policy to improve the energy efficiency (EE) of heterogeneous cellular networks. The combination of active and inactive base stations (BS) that allows for maximizing EE is identified as a combinatorial learning problem and requires high computational complexity as well as a large signaling overhead. This paper aims at presenting a learning policy that dynamically switches a BS to ON or OFF status in order to follow the traffic load variation during the day. The network traffic load is represented as a Markov decision process, and we propose a modified upper confidence bound algorithm based on restless Markov multi-armed bandit framework for the BS switching operation. Moreover, to cope with initial reward loss and to speed up the convergence of the learning algorithm, the transfer learning concept is adapted to our algorithm in order to benefit from the transferred knowledge observed in historical periods from the same region. Based on our previous work, a convergence theorem is provided for the proposed policy. Extensive simulations demonstrate that the proposed algorithms follow the traffic load variation during the day and contribute to a performance jump-start in EE improvement under various practical traffic load profiles. It also demonstrates that proposed schemes can significantly reduce the total energy consumption of cellular network, e.g., up to 70% potential energy savings based on a real traffic profile.

本文提出了一种学习策略，以提高异构蜂窝网络的能效（EE）。允许最大化EE的活动基站和非活动基站（BS）的组合被识别为组合学习问题，并且需要较高的计算复杂度和较大的信令开销。本文旨在提出一种学习策略，可以动态地将BS切换为ON或OFF状态，以便跟踪白天的流量负载变化。网络流量负载被表示为马尔可夫决策过程，我们提出了一种基于躁动马尔可夫多臂强盗框架的改进的上置信界算法，用于基站交换操作。此外，为了应对最初的奖励损失并加快学习算法的收敛速度，迁移学习概念适用于我们的算法，以便从历史时期从同一地区观察到的迁移知识中受益。根据我们以前的工作，为拟议的政策提供了一个收敛定理。大量的仿真表明，所提出的算法可跟踪白天的交通负荷变化，并有助于在各种实际交通负荷情况下提高EE的性能。它还表明，提出的方案可以显着降低蜂窝网络的总能耗，例如，基于实际流量概况，可节省多达70％的潜在能耗。大量的仿真表明，所提出的算法可跟踪白天的交通负荷变化，并有助于在各种实际交通负荷情况下提高EE的性能。它还表明，提出的方案可以显着降低蜂窝网络的总能耗，例如，基于实际流量概况，可节省多达70％的潜在能耗。大量的仿真表明，所提出的算法可跟踪白天的交通负荷变化，并有助于在各种实际交通负荷情况下提高EE的性能。它还表明，提出的方案可以显着降低蜂窝网络的总能耗，例如，基于实际流量概况，可节省多达70％的潜在能耗。