Integrating reinforcement learning and skyline computing for adaptive service composition

Abstract

In service computing, combining multiple services through service composition to address complex user requirements has become a popular research topic. QoS-aware service composition aims to find the optimal composition scheme with the QoS attributes that best match user requirements. However, certain QoS attributes may continuously change in a dynamic service environment, so service composition methods need to be adaptive. Furthermore, the large number of candidate services poses a key challenge for service composition, where existing service composition approaches based on reinforcement learning (RL) suffer from low efficiency. To deal with the problems above, in this paper, a new service composition approach is proposed which combines RL with skyline computing where the latter is used for reducing the search space and computational complexity. A WSC-MDP model is proposed to solve the large-scale service composition within a dynamically changing environment. To verify the proposed method, a series of comparative experiments are conducted, and the experimental results demonstrate the effectiveness, scalability and adaptability of the proposed approach.

Introduction

Service composition is a widely used software engineering paradigm to build complex and value-added software [1], [2]. Due to its inter-operability, reusability, deployability, service composition has become one of the key technological choices to deal with complex user requirements by combining multiple atomic services [3]. Services provided by multiple service providers usually have different Quality of Service (QoS), such as price, reliability, reputation, throughput, and response time. In service composition, corresponding QoS constraints need to be considered, leading to QoS-aware service composition, which aims to generate optimal or near-optimal composite services that meet user requirements.

In a dynamic service environment, certain QoS attributes may continuously evolve. As a result, service composition method needs to adapt to the changing environment. Moreover, things may become more complicated if a service composition involves complex service workflows (e.g., WSC-MDP in Fig. 2) and a large number of candidate services, which becomes common for enterprise applications [4]. Nowadays, there are a large number of services on the Internet. For example, programmableweb.com¹ has documented over 22,770 APIs by September 2019, and the number of APIs is growing at an alarming rate every year. Thus, efficiency is an important and urgent aspect that must be taken into account. To solve large-scale problems, there are some existing methods like multi-agent in [5], multi-level index technology in [6]. In [5], multiple agents work together and speed up the convergence rate of the algorithm; In [6], authors propose a multi-level index model to expedite Web service discovery and composition. In general, these two methods do not reduce the number of candidate services, so there will be some unnecessary explorations in the process of learning. In this paper, we utilize skyline computing [7], [8] to address the above limitations. Because the skyline chooses high-quality services from a large candidate pool, it can significantly reduce the search space, leading to efficient computation.

To deal with a dynamic environment, we leverage the advantage of reinforcement learning (RL), which learns by trial-and-error interactions with the dynamic environment and thus has good adaptability. Thus, introducing the RL into the process of service composition can optimize the service composition solution and adapt to the dynamic environment. RL is a major type of machine learning method that has become a useful technique to solve sequential decision making problems [9]. In an RL system, a learning agent learns an optimal policy via interactions with an uncertain environment. In each step, the learning agent chooses and executes the optimal action to maximize the long-term reward, instead of being told which action to take. Afterwards, the agent receives a scalar reward and the current state transits to its successive state. Finally, the agent evaluates the effect of this state transition.

In the context of service composition, on the one hand, the environment is constantly changing and certain QoS may continuously evolve. On the other hand, there exist increasing complex composition flows and a huge number of candidate services. Hence, how to adapt to dynamic environment and how to achieve high efficiency are nontrivial. In order to cope with the above two challenges, we combine reinforcement learning and skyline computing in this paper. RL is to respond to dynamic environment and achieve good adaptability. Skyline computing is used to reduce the search space and improve efficiency. More specifically, skyline computing extracts data points which are not dominated by any other point on all QoS dimensions.

In this paper, we develop a service composition approach that combines RL with skyline computing. The main contributions are summarized as follows:

• In the process of service composition, we present a new method to reduce the search space and computational complexity by exploiting skyline computing.

• A WSC-MDP model is designed to solve the large-scale service composition problems, which can also deal with a dynamically changing environment.

• We conduct a series of experiments to demonstrate the effectiveness, scalability and adaptability of the proposed approach.

Table 1 summarizes the main notations used in the rest of the paper. The remainder of this paper is organized as follows. Section 2 gives an overview of related work. Section 3 presents the preliminaries that lay the foundation of the proposed approach. Section 4 introduces the Web Service Composition Markov Decision Process (WSC-MDP) model and service composition algorithm. Section 5 details the experimental evaluation and comparison with other related works. Finally, Section 6 draws our conclusions and identifies some future directions.