End-to-end CNN-based dueling deep Q-Network for autonomous cell activation in Cloud-RANs

Highlights

• A framework for autonomous cell activation/deactivation and customized physical resource allocation in C-RANs.

• An end-to-end CNN-based relational dueling DQN approach for autonomous cell activation.

• A customized physical resource allocation scheme for rate and delay to balance QoS satisfaction and resource utilization with minimum number of active RRHs.

• A comprehensive set of simulation results is provided.

Abstract

The fifth generation (5G) technology is expected to support a rapid increase in infrastructure and mobile user subscriptions with an increase in the number of remote radio heads (RRHs) per unit area using cloud radio access networks (C-RANs). From the economic point of view, minimizing the amount of energy consumption of the RRHs is a challenging issue. From the environmental point of view, achieving “greenness” in wireless networks is one of the many goals of telecommunication operators. This paper proposes a framework to balance the energy consumption of RRHs and quality of service (QoS) satisfaction of users in cellular networks using a convolutional neural network (CNN)-based relational dueling deep Q-Network (DQN) scheme. Firstly, we formulate the cell activation/deactivation problem as a Markov decision process (MDP) and set up a two-layer CNN which takes raw captured images in the environment as its input. Then, we develop a dueling DQN-based autonomous cell activation scheme to dynamically turn RRHs on or off based on the energy consumption and QoS requirements of users in the network. Finally, we decouple a customized physical resource allocation for rate-constrained users and delay-constrained users from the cell activation scheme and formulate the problem as a convex optimization problem to ensure the QoS requirements of users are achieved with the minimum number of active RRHs under varying traffic conditions. Extensive simulations reveal that the proposed algorithm achieves faster rate of convergence than nature DQN, Q-learning and dueling DQN schemes. Our algorithm also achieves stability in mobility scenarios compared with DQN and dueling DQN without CNN. We also observe a slight improvement in balancing energy consumption and QoS satisfaction compared with DQN and dueling DQN schemes.

Introduction

The rapid surge in the number of connected mobile devices from varying applications in recent times has prompted infrastructure providers and service operators to expand their infrastructural network assets to be able to match the increased number of subscriptions. A research paper published by the international telecommunication union (ITU) in 2016 indicates that the mobile and wireless data traffic density in the world will increase by a 1000 fold in 2025 (ICT Data and Statistics Division, 2016). This has led to much attention being turned to research on energy consumption issues in the fifth generation (5G) cellular networks. As the issue of global warming is increasing day-by-day, telecommunication regulatory bodies such as 3rd generation partnership project (3GPP), ITU, etc. are seeking to design green cellular networks (Telecommunication Management, 2010). This motivation has given birth to innovative projects like the EARTH project and the GreenTouch project (Correia et al., 2010). Existing researches have revealed that base stations (BSs) are responsible for 60%–80% of the total energy consumption in cellular networks (Marsan et al., 2009) and the traffic load on BSs is less than one-tenth of the peak value for 30% time of weekdays (Oh et al., 2011). This calls for the deactivation of BSs dynamically, in which BSs automatically enter sleep mode when the traffic volume is low to ensure low energy consumption.

In traditional radio access networks (RANs), the baseband processing units are installed at each small cell BS. However, the deployment of small cells in RANs causes high energy consumption and increases the computational burden heavily (Cai et al., 2016). Cloud radio access network (C-RAN) is a promising energy-efficient network architecture in 5G using cell activation techniques. In C-RANs, the baseband units (BBUs) are installed in the cloud, rather than at small cells. In this way, C-RANs have the prospect of reducing the capital expenses (CapEx) and operational expenses (OpEx) of network operators. All signal processing and upper layer functionalities are performed by the BBU pool in the cloud and the remote radio heads (RRHs) are responsible for radio frequency transmission and reception (Luo et al., 2015). The centralized structure of C-RANs allows RRHs to be controlled by a common BBU pool, which can switch the RRHs on or off to reduce energy consumption. C-RANs can also reduce interference and improve cooperative processing gain through cloud computing techniques, thereby solving the energy efficiency (EE) problem in next generation networks. In an attempt to provide an energy-efficient cellular network, many researchers have formulated the joint cell activation, user association and resource allocation problem as convex optimization problems with different objective functions (Mesodiakaki et al., 2014; Xu et al., 2014; Zhuang et al., 2016a; Koudouridis et al., 2012a). However, the traditional model-based solutions may be impractical in dynamic wireless network environments where mobility of users affect the network statistics at each time frame. In this case, model-free solutions are likely to achieve gains since they optimize the network EE in the entire operational period in real-time. Reinforcement learning (RL) is a model-free machine learning technique with a centralized learning agent who can tackle complex decision-making problems and manage the energy consumption of RRHs based on the current state. In the scenario of energy-efficient C-RAN architecture, the learning agent selects possible actions in each state and then trains this data to make a decision whether to turn RRHs on or off at each decision epoch.

In recent times, deep learning (DL) has been successfully applied in speech recognition, and natural language processing, e.g. convolutional neural networks (CNN) (Lecun et al., 2015). DL has also been proposed in wireless communication problems to help RL algorithms learn sequential control tasks in end-to-end fashion (Cheng et al., 2017a, 2017b). CNN has the ability to capture more complex dynamics in mobility scenarios. Most existing machine learning algorithms such as Q-learning and deep reinforcement learning (DRL) applied in wireless network problems define the states of the users with hand-crafted features. If the relationship between users and RRHs are neglected, feature extraction may be somehow artificially operated and the learning agent may achieve sub-optimal solutions. On the other hand, users will have to report their information to their respective RRHs, which may result in a big burden on signaling overhead as feedback. With these shortfalls in mind, we introduce a two-layer end-to-end CNN-based relational dueling deep Q-Network (DQN) method with randomly-captured environment states. The raw observations are in the form of an “image” capturing the user-RRH relationships in the network. The main difference between the traditional DRL (nature DQN) and our proposed CNN-based relational dueling DQN method is that, traditional DRL develops a saturated replay memory with experiences from the current time frame under mobility scenarios. This may result in sub-optimal solutions from the learning agent. The DRL-based scheme in Xu et al. (2017) omits vital statistical characteristics of the users in the network which may have an impact on the outcome of the learning process. Our proposed scheme combines CNN and dueling DQN taking the raw image features between users and RRHs as raw input observations from the network. The CNN phase carries out the feature extraction process and the dueling DQN phase trains the network to achieve faster convergence. Dueling DQN ignores unnecessary states in the learning process, which enables faster rate of convergence. Also, we define a multi-objective reward function with coefficients to indicate the importance of each metric in the reward. To efficiently solve the energy consumption problem, we first formulate the cell activation problem as a Markov decision process (MDP) where we define states, actions, reward and future states. Then we develop an autonomous cell activation scheme based on CNN and dueling DQN techniques to activate or deactivate RRHs with respect to the energy consumption and quality of service (QoS) satisfaction requirements of users. Finally, we formulate the physical resource allocation problem as a convex optimization problem and solve to balance energy consumption and QoS satisfaction of users with the minimum number of active RRHs. The main contributions of this paper are as follows:

• We propose an autonomous decision-making framework for cell activation based on DL and DRL methods to minimize energy consumption of RRHs and ensure the QoS satisfaction of users in C-RANs.

• We formulate the cell activation problem as an MDP and set up a two-layer end-to-end CNN architecture that takes raw images captured by the RRHs as input states. The output of the CNN is fed to the dueling DQN as input to ensure autonomous cell activation or deactivation considering varying traffic profile and user mobility. The CNN part of the algorithm extracts image features of the network environment and the dueling DQN part has the ability to determine the best action for each state, making the learning process faster.

• We formulate the EE-QoS optimization problem as a convex optimization problem and customize it for rate-constrained users and delay-constrained users, but unified with aggregated QoS satisfaction.

• Comprehensive simulations are conducted to show that the proposed algorithm has a better performance in terms of balancing the energy consumption of RRHs and QoS satisfaction of users. The proposed algorithm also achieves faster rate of convergence and stable performance in dynamic mobility scenario.

The rest of the paper is organized as follows: Section 2 presents the related works and Section 3 describes the system model in terms of network model, traffic model, power consumption model and utility model. Section 4 introduces the problem formulation for autonomous energy management. Furthermore, we present the two-layer CNN-based relational dueling DQN algorithm in Section 5. Section 6 presents the simulation results and analysis. Finally, the conclusion is presented in Section 7.