Distributed learning dynamics of Multi-Armed Bandits for edge intelligence

Abstract

Multi-agent decision making is a fundamental problem in edge intelligence. In this paper, we study this problem for IoT networks under the distributed Multi-Armed Bandits (MAB) model. Most of existing works for distributed MAB demand long-time stable networks connected by powerful devices and hence may not be suitable for mobile IoT networks with harsh IoT constraints. To meet the challenge of resource constraints in mobile IoT environment, we propose a lightweight and robust learning algorithm in a dynamic network allowing topology changes. In our model, each agent is assumed to have only limited memory and communicate with each other asynchronously. Moreover, we assume that the bandwidth for exchanging information is limited and each agent can transmit $O\left({log}_{2}K\right)$ bits ($K$ denotes the number of arms) per communication. Rigorous analysis shows that despite these harsh constraints, the best arm/option can be identified collaboratively by the agents and the algorithm converges efficiently. Extensive experiments illustrate that the proposed algorithm exhibits good efficiency and stability in mobile settings.

Introduction

Recent years have witnessed the prosperity of internet of things (IoT) due to the rapid advancement of technology of chip integration, wireless communication, big data and artificial intelligence, etc. IoT devices, which distributed at the network edge, make data explored efficiently and facilitate the data aggregation through local computation and communication with each other. With these merits, IoT seems to be a good fit for implementing large scaled machine learning algorithms and developing edge intelligence, which in turn helps to complete some tasks in IoT such as decision making, underlying environment learning, etc. Hence the learning paradigm shifts from centralized cloud computing to edge computing (fog computing) recently which results in edge intelligence. Edge intelligence refers to a set of connected edge devices (e.g., IoT devices, smartphones, UAVs) that is used for data collection, caching, processing, and analysis based on artificial intelligence [1]. However, traditional machine learning algorithms are typically demanding in computing resources [2], [3], e.g. CPU, GPU, and memory, which contrasts the lightweight computation capacity, limited memory and mobility of edge devices. To push the computation tasks from the cloud server to the edge, we have to devise lightweight learning algorithms [4], [5], [6] for the edge (e.g. IoT networks).

Multi-agent decision making is a common task in edge intelligence. We employ distributed multi-armed bandits (MAB) to formalize our multi-agent decision-making problem, where multiple networked agents face a set of arms (options) and each arm is represented by a stochastic reward with a mean unknown to the agent. The agents’ goal is to learn the best arm/option collaboratively by exchanging local states or opinions between each other iteratively until reaching a consensus. In fact, distributed MAB algorithm for large scaled networks has been studied extensively (see Section 2). Unfortunately, most of these works demand long-time stable networks connected by powerful devices, which are not suitable for edge intelligence. A natural question comes that whether an efficient and robust collaborative learning algorithm can be devised such that it can overcome constraints of IoT devices and be implemented even in mobile IoT networks.

To settle this problem, we present in this paper a collaborative learning algorithm for dynamic IoT networks. In particular, the network is modeled as a connected network whose topology changes arbitrarily over time on a fixed set of agents [7], [8], [9], [10]. For the sake of being suitable for IoT, we also impose some necessary constraints on the system model. Firstly, we assume that each agent has only bounded memory such that no learning history can be stored. Secondly, the bandwidth of communication channel is limited such that each agent can transmit at most ${log}_{2}K$ ($K$ denotes the number of arms) bits per communication. Lastly, the time is assumed to be continuous and agents take actions asynchronously by equipping each agent an independent Poisson clock with a common parameter. The Poisson clock model is natural for modeling swarm intelligent systems that are not fully synchronized [11], [12].

Under the dynamic network model we mentioned above, we propose the distributed collaborative learning algorithm that can be implemented in dynamic networks under the IoT constraints. With detailed and rigorous analysis, it can be assured with a high probability that every agent pulls only the best arm eventually. We also conducted extensive experiments to confirm the efficacy of our proposed distributed learning algorithm. The results show that the convergence of our learning algorithm is quick and hence the algorithm is robust and efficient in real settings.

The remainder of this paper is organized as follows. Section 2 introduces the related works. Section 3 presents our system model. Section 4 and Section 5 show the learning algorithm. We give the detailed analysis in Section 6. Section 7 shows the practical performance evaluation. Finally we conclude our paper in Section 8.