Towards cost-effective service migration in mobile edge: A Q-learning approach

https://doi.org/10.1016/j.jpdc.2020.08.008Get rights and content

Highlights

  • A Q-learning-based framework is proposed for service migration in mobile edges.

  • A policy-cache-based method is introduced to optimize the learning process of the framework.

  • A software-defined approach is presented to effectively implement the framework.

  • A prototype implementation is evaluated with the trace data gathered from Shenzhen taxi system and metro stations.

Abstract

Service migration in mobile edge computing is a promising approach to improving the quality of service (QoS) for mobile users and reducing the network operational cost for service providers as well. However, these benefits are not free, coming at costs of bulk-data transfer, and likely service disruption, which could consequently increase the overall service costs. To gain the benefits of service migration while minimizing its cost across the edge nodes, in this paper, we leverage reinforcement learning (RL) method to design a cost-effective framework, called Mig-RL, for the service migration with a reduction of total service costs as a goal in a mobile edge environment. The Mig-RL leverages the infrastructure of edge network and deploys a migration agent through Q-learning to learn the optimal policy with respect to the service migration status. We distinguish the Mig-RL from other existing works in several major aspects. First, we fully exploit the nature of this problem in a modest migration space, which allows us to constrain the number of service replicas whereby a defined state–action space could be effectively handled, as opposed to those methods that need to always approximate a huge state–action space for policy optimality. Second, we advocate a migration policy-base as a cache to save the learning process by retrieving the most effective policy whenever a similar migration pattern is encountered as time goes on. Finally, by exploiting the idea of software defined network, we also investigate the efficient implementation of Mig-RL in mobile edge network. Experimental results based on some real and synthesized access sequences show that Mig-RL, compared with the selected existing algorithms, can substantially minimize the service costs, and in the meantime, efficiently improve the QoS by adapting to the changes of mobile access patterns.

Introduction

Mobile edge computing (MEC) is a new computing paradigm that integrates the advantages of both mobile computing and edge computing to improve the quality of service (QoS) for mobile users. In particular, with MEC, computing resources could be pushed from cloud center to its network edge, which allows data services as well as other related processing tasks to run in proximity of mobile users. Consequently, it not only reduces the service latency but also minimizes the network traffic, both benefits are fairly important to those time-bounded services, which are typical applications in mobile computing.

Given its access efficiency and cost reduction, MEC is quickly becoming a spotlight in the next wave of cloud research, especially with the advance of 5G networks [4], [10], where more intelligent and powerful computing resources can be installed at the edge of wireless network [10], [25], enabling many time-sensitive services to be deployed close to users. Although the advantage of MEC is apparent, it still suffers from the inefficiency in serving users in mobility as the requests made by the mobile users are always changing with respect to time and location, and the limited coverage of edge servers often results in the failure to comply with the requirements on access latency. As such, the provided services in short of considering these factors may significantly increase the access delays and, much worse, impose a large amount of network traffic, leading to service disruption and performance degradation.

To mitigate this problem, one of the covet and challenging technologies is Service Migration, which means a service at the edge can be migrated in a form of virtual machine or container from one edge server to another as a self-adaptive method to the user mobility for more effective processing in proximity. Since a service is typically accessed by a large number of mobile users at different time and locations, its migration in general cannot be designed for each individual user, instead, it is usually relied on the dynamic changes of mobile access patterns in spatial–temporal dimensions. As with other often-used optimization technologies, the service migration is also not free, coming at costs of bulk-data transfer, and likely service disruption, which as a result increases the overall service costs. As such, it will be a great challenge to make judicious migration decisions for various settings, and as well develop an efficient way to implement it in mobile edge network.

Reinforcement learning (RL) [26], as a sort of machine learning algorithms, has been applied to many large-scale optimization problems in different application fields. In the algorithm, a decision maker or agent perceive the environment and chooses an action at each state. As a reaction, the environment would feedback a value as a reward after each action, indicating to the agent the quality of the applied action. The final goal of the agent is to learn a policy that selects the best sequence of actions to maximize the accumulated rewards.

In this paper, we study the service migration problem in a mobile edge network and propose a RL-based migration framework, called Mig-RL, based on the Q-learning algorithm—a one-step off-policy algorithm for temporal difference learning [26]. The rationales behind this design are twofold. On the one hand, the nature of the migration problem is to determine a sequence of migration policies for the goal in time series, which can be modeled as Markov Decision Process (MDP) to fit the RL-based algorithms, on the other hand, the deployment of this algorithm in the mobile edge network is relatively simple, flexible, and efficient given the mobile network infrastructure. With the Mig-RL, a learning agent could learn the online migration policy through the interactions with environments, which determines when to switch a virtual machine (VM) that hosts the service into an active migration state, and further finds the next best stop in a certain probability, according to the currently served demands. The Mig-RL is self-optimized and characterized by the effective use of gathered access information to conduct migration that overcome the limitation of traditional local search in cost reduction [20].

In particular, we distinguish the Mig-RL from other existing works in several major aspects. First, we fully exploit the nature of this problem in a modest migration space (i.e., city-wide area), which allows us to constrain the number of service replicas whereby a defined state–action space could be efficiently handled, as opposed to those methods that always need to approximate the huge state–action space for policy optimality at cost of large computation overhead, like the neural network training in deep reinforcement learning (DRL) [26]. This design is reasonable as our previous studies show that a small number of servers are always sufficient in a relatively large mobile environment to serve dynamic service demands in a cost effective way [29], [31]. As a result, the size of Q-Table in the algorithm could be considerably reduced to handle the problem with a modest state–action space.

Second, to get rid of the policy computation for some states, we advocate a migration policy-base as a cache to save the learning process by retrieving the most effective policy from the cache whenever a similar migration state is encountered as time goes on. With this design, the advantages of the Q-learning-based algorithm and the infrastructure of the network can be fully and efficiently exploited to mitigate the cold-start effects and implement highly cost-effective migration service.

Third, by exploiting the idea of software defined network (SDN) to enable programmatically efficient network configuration for network performance and monitoring [13], we also investigate the efficient implementation of Mig-RL for cost-effective service migration in the mobile edge network. Specifically, we first decouple the movement process of the service from the migration control process, and then map a decision agent MIG-Agent in the framework to a migration controller, a centralized server deployed either in the MEC clusters or in an appropriate place in the edge network to facilitate the migration control. In turn, the decision agent sends the control processes to the corresponding edge servers associated to one or more Base Stations (BSs) in the network.

Finally, we showed that, via empirical studies, the proposed algorithm exhibits better performance in service migration by adapting to the changes of mobile access patterns in a cost-effective way.

The remainder of this paper is organized as follows. We introduce the background knowledge of mobile edge network and Q-learning in Section 2.2. We introduce the service migration problem, together with its RL-based algorithm, Mig-RL, in Section 3. After that, we present the implementation of the Mig-RL framework in Section 5, followed by the simulation results in Section 6. We review some related work in Section 7 for comparison study and conclude the paper in the last section.

Section snippets

Background knowledge

In this section, we review some background knowledge that is helpful to understand this paper. We first describe a typical infrastructure of the mobile edge network for which our migration framework is designed, and then introduce some basic knowledge regarding the reinforcement learning in general and Q-learning in particular. Finally, how the service migration is also managed via learning algorithm in this network is also discussed.

Service migration model

We consider an arbitrary n-node network G(V,E) in mobile edge as a service infrastructure (see Fig. 1) where the service is running in k virtual servers (hereafter servers in short). The servers can be hosted by any set of compute nodes (aka MEC servers), and are accessed by a sequence of batch requests issued from external machines (i.e., mobile terminals). The requests arrive in an online order and are served in turn by triggering the migration of the servers. As a result, the locations of

RL-based migration algorithm

We are now describing our RL-based migration algorithm, called Mig-RL, for k server migration in the edge network to minimize the cumulative service cost. Based on the described migration model, we can fit the service migration task within the agent environment framework where the agent is a decision maker, namely MIG-Agent, that keeps monitoring the access workloads of each server, and in turn, makes intelligent migration decision to adapt the k servers to the dynamics of the environment with

Implementation framework

After understanding the design of the Mig-RL algorithm, we delve into its implementation framework, which is built on top of a MEC cluster shown in Fig. 5. In reality, this cluster can be composed of a collection of the inter-connected MEC servers, together with the MME, that are majorly deployed in the RAN segment of the 4G network infrastructure as shown in Fig. 1. We first present optimized framework that can strike a balance between the algorithm efficiency and the policy optimality, and

Empirical studies

We conducted experiments to show how the proposed Mig-RL algorithm brings in benefits by migrating servers as an adaptation to the mobile accesses to a cloud service. To reach this goal, we deliberately leveraged the taxi transaction data on May 20, 2014, totally 50MB gathered from 15 hot areas in Shenzhen, China, to simulate the mobile requests for the service [24] as the true request data is not available due to the privacy issue. The rationale behind this consideration is that most

Related work

Given its importance in utilization of resources, savings of energy consumption, and fault tolerance of host machines, service migration has been drawing great attention from both academia and industry to promote the efficiency of cloud computing in recent years [16], [19], [21], [31]. Mishra et al. [19] presented the virtual machine migration techniques and their usage towards dynamic resource management in virtualized environments while Phan et al. [21] adopted dynamic service migration

Conclusions

In this paper, we formulated and studied the service migration problem in mobile edge network so that the mobile accesses to the service could be adaptively satisfied with minimum cost. To this end, we developed an efficient algorithm, called Mig-RL, based on Q-learning approach for k server migration in an n-node network. The algorithm is characterized by its distinct definitions of the state penalty score, and in turn, the action reward, that are exploited to conduct multi-step migration to

CRediT authorship contribution statement

Yang Wang: Conceptualization, Methodology, Writing - original draft. Shan Cao: Implementation of the baseline. Hongshuai Ren: Analysis and design. Jianjun Li: Software. Kejiang Ye: Experiments. Chengzhong Xu: Writing - review & editing. Xi Chen: Proof reading.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Yang Wang received the B.Sc. degree in applied mathematics from Ocean University of China, in 1989, and the M.Sc. computer science from Carleton University, in 2001, and the Ph.D degree in computer science from the University of Alberta, Canada, in 2008. He is currently in Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, as a professor. His research interest includes cloud computing, big data analytics, and Java virtual machine on multicores. He is an Alberta Industry

References (34)

  • F. Farahnakian, P. Liljeberg, J. Plosila, Energy-efficient virtual machines consolidation in cloud data centers using...
  • Z. Gao, Q. Jiao, K. Xiao, Q. Wang, Z. Mo, Y. Yang, Deep reinforcement learning based service migration strategy for...
  • Y.C. Hu, M. Patel, D. Sabella, N. Sprecher, V. Young, Mobile edge computing—A key technology towards 5G, ETSI White...
  • M. Khelghatdoust, V. Gramoli, D. Sun, GLAP: Distributed Dynamic Workload Consolidation through Gossip-Based Learning,...
  • KimS.

    One-on-one contract game–based dynamic virtual machine migration scheme for Mobile Edge Computing

    Trans. Emerg. Telecommun. Technol.

    (2017)
  • KreutzD. et al.

    Software-defined networking: A comprehensive survey

    Proc. IEEE

    (2015)
  • KyungY. et al.

    Software defined service migration through legacy service integration into 4G networks and future evolutions

    IEEE Commun. Mag.

    (2015)
  • Cited by (10)

    • Security-Aware and Time-Guaranteed Service Placement in Edge Clouds

      2023, IEEE Transactions on Network and Service Management
    • PLCD: Policy Learning for Capped Service Mobility Downtime

      2023, Proceedings - International Conference on Computer Communications and Networks, ICCCN
    • Machine Learning for Service Migration: A Survey

      2023, IEEE Communications Surveys and Tutorials
    • Survey of Data-Driven Application Self-Adaptive Technology

      2022, Jisuanji Yanjiu yu Fazhan/Computer Research and Development
    • DT-EEC: A Digital Twin-assisted End-Edge-Cloud Collaboration Architecture for Industrial Internet

      2022, Proceedings - 2022 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Autonomous and Trusted Vehicles, Scalable Computing and Communications, Digital Twin, Privacy Computing, Metaverse, SmartWorld/UIC/ATC/ScalCom/DigitalTwin/PriComp/Metaverse 2022
    View all citing articles on Scopus

    Yang Wang received the B.Sc. degree in applied mathematics from Ocean University of China, in 1989, and the M.Sc. computer science from Carleton University, in 2001, and the Ph.D degree in computer science from the University of Alberta, Canada, in 2008. He is currently in Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, as a professor. His research interest includes cloud computing, big data analytics, and Java virtual machine on multicores. He is an Alberta Industry R&D Associate (2009–2011), and a Canadian Fulbright Scholar (2014–2015).

    Shan Cao received her B.E. degree in Software Engineering from Dalian University of Technology in 2015, and the M.Sc. degree in Computer Science from University of Chinese Academy of Sciences in 2018. She is currently working as a Fintech Management Trainee in China Merchants Bank. Her research interests include reinforcement learning, cloud computing and big data analytics.

    Hongshuai Ren received his B.Sc. degree in Software Engineering from Jilin Engineering Normal University in 2016, and M.S. degree in Software Engineering from Northeast Normal University, in 2019. He is currently studying at Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences. His research interests include deep learning and resource scheduling in cloud computing.

    Jianjun Li received the B.Sc. degree in information engineering from Xi’an University of Electronic Science and Technology, Xi’an, China, and the M.Sc. and Ph.D degrees in electrical and computer from The University of Western Ontario and University of Windsor, Canada separately. He is currently working at Hangzhou Dianzi University as a chair professor. His research interests include machine learning algorithms and implementation in cloud platforms.

    Kejiang Ye received his B.Sc. and Ph.D. degree in Computer Science from Zhejiang University in 2008 and 2013 respectively. He was also a joint Ph.D. student at The University of Sydney from 2012 to 2013. After graduation, he works as Post-Doc Researcher at Carnegie Mellon University from 2014 to 2015 and Wayne State University from 2015 to 2016. He is currently an Associate Professor at Shenzhen Institutes of Advanced Technology, Chinese Academy of Science. His research interests focus on the performance, energy, and reliability of cloud computing and network systems.

    Chengzhong Xu received the Ph.D. degree from the University of Hong Kong in 1993. He is currently the Dean of Faculty of Science and Technology, University of Macau, China. His research interest includes parallel and distributed systems and cloud computing. He has published more than 200 papers in journals and conferences. He serves on a number of journal editorial boards, including IEEE TC, IEEE TPDS, IEEE TCC, JPDC and China Science Information Sciences. He is a fellow of the IEEE.

    Xi Chen received the B.Sc. degree in geographic science from Xinjiang University, China, in 1985, the M.Sc. from the Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences (CAS), in 1988, and the Ph.D degree in geographic science from Wuhan University, China, in 2003. He is currently the Director of Research Center for Ecology and Environment of Central Asia, CAS. His research interest is in database and big data processing, spatial information systems. He has published more than 100 technical papers in related conferences and journals.

    This document is the results of the research project funded in part by Key-Area Research and Development Program of Guangdong Province (2020B010164002, 2019B010137002), and also in part by Science and Technology Development Fund of Macao SAR (FDCT) (1058No.0015/2019/AKP), the Research Center for Ecology and Environment of Central Asia, Chinese Academy of Sciences (SQ2016YFHZ020520), National Natural Science Foundation of China (61672513, 61871170), and Shenzhen Basic Research Program (JCYJ20170818153016513). The second and third authors contributed equally to this work.

    View full text