Modeling the UE-perceived cellular network performance following a controller-based approach

Abstract

During the last few years, mobile communication networks have experienced a huge evolution. This evolution culminates with the arrival of the fifth generation (5G) of mobile communication networks. As a result, the complexity of network management tasks has been increasing and the need to use automatic management algorithms has been demonstrated. However, many mobile network operators (MNOs) are reluctant to evaluate these algorithms in their networks. To address this issue, in this paper, a modeling approach is proposed. In this sense, the behavior of a commercial network, as it is perceived by user equipments (UEs), has been replicated in a research testbed using a three-step modeling process. The first step consists on performing a measurement campaign in several external networks. The second step is composed of the measurement campaign result analysis and the classification of the results in different types of scenarios. Finally, the third step is related to the application of a modeling algorithm in a research testbed. In order to perform the last step, the use of a method based on a controller is proposed. The modeling process presented in this paper allows to replicate the network behavior from users located in different areas and with different conditions point of view. Moreover, the use of a testbed environment can help to avoid downtime in commercial networks caused by possible algorithm bugs.

1 Introduction

Mobile communication networks have experienced a large development over the last few years. According to studies conducted by Cisco [1], “Global mobile data traffic was 7% of total IP traffic in 2016 and will be 17% of total IP traffic by 2021”. This growth in traffic will be especially pronounced with the arrival of the fifth generation (5G) of mobile communication networks. In these networks, an increment in the number of users and offered services is also expected, resulting in an increase in the number of configuration parameters of the networks and possible scenarios. In this context, the development of automatic algorithms that allow the management of networks in a faster and more efficient way become a priority for mobile network operators (MNOs). However, MNOs are increasingly reluctant to implement in their own network algorithms that have not been assessed in prototypes of real networks. The reason is to avoid a possible service degradation due to a suboptimal parameter configuration change made by a certain algorithm. This problem is getting worse in 5G networks, since in these networks, a reduction of the downtime is sought. Thus, many automatic management network algorithms developed by researchers and tested in simulated environments are never implemented in real networks. In the cases in which the MNOs decide to assess an algorithm in their own networks, the tests will be carried out in limited areas of the network before extending them to the complete network. This methodology presents some important limitations, since different network areas may behave in a different way and, therefore, the conclusions obtained from a specific area of the network might not be generalizable to the entire network.

To deal with this problem, a modeling approach could be followed. Modeling is the act of representing something. This representation is often mathematical. Thus, in the literature, there are many works focused on the development of analytical models that describe the behavior of different types of phenomena. In relation to mobile communication networks, in [2] and [3], the authors present analytical models of the behavior of different metrics associated with the quality of experience (QoE) of video services. Download throughput models are proposed in [4] and [5]. In the first case, the authors use neural networks, whereas in the second case, the authors use linear regression. However, modeling can also be the act of imitating the behavior of a specific situation. In mobile communication network scope, to the authors’ knowledge, in the literature, there are no other works related to the modeling in this line.

On the other hand, with the aim of satisfying the needs of a growing number of users, the goal of network management tasks has been changing over time, from management focused on the performance of a certain section of the network (typically the radio access network), such as in [6], to a management that is increasingly closer to the user, and to the user’s perception of the performance or behavior of the network [7], obtaining in this way an end-to-end (E2E) vision of the network behavior.

In line with the latter, this paper aims at showing how the behavior of a commercial cellular network could be replicated from a user equipment (UE) point of view by using a research testbed (Fig. 1). In this sense, by the adjustment of the configuration parameters of the testbed network, it is intended to obtain a network equivalent to the commercial one in terms of user perception, that is, to ensure that users connected to the testbed network report the same values of users’ performance indicators (measured user performance indicators in Fig. 1) than the users connected to the commercial network (target user performance indicators in Fig. 1), thus providing an environment in which testing automatic management network algorithms before their implementation in MNO commercial networks. One of the main advantages of the use of a network model is that different areas, with different behaviors, could be independently modeled and optimized. In addition, the use of a testbed environment allows detecting possible failures or bugs in the algorithms before being implemented in the real network, avoiding possible configuration failures of the network parameters and the consequent service degradations.

Fig. 1

Modeling scheme

To achieve the global objective, the modeling of a cellular network behavior from a UE perspective, a three-step process is proposed. The first step consists of performing a measurement campaign. For this purpose, similarly to the measurement campaign conducted in [8], experiments have been carried out in commercial networks of some of the main MNOs in Europe. The second step consists in the analysis and classification of the data collected in the measurement campaign, obtaining different behavior patterns of the measured networks. The output of the second step is taken as the input to perform the last step. Such step refers to the modeling of a commercial network performance from the UE point of view. In order to perform the modeling, a method based on a controller has been used. In particular, the use of the Taguchi method (TM) is proposed, in a similar way to the one proposed in [9], where antenna azimuths and tilts are optimized in order to improve the user throughput. In this paper, the TM controller aims at adjusting the value of specific network configuration parameters depending on the behavior to be replicated that will be represented by user performance indicators.

The rest of the paper is organized as follows. Firstly, the methodology developed in the modeling process is presented, indicating how the design of the measurement campaign has been made and providing the main characteristics of the proposed method to carry out the modeling. Secondly, an overview of the testbed network is provided, and the result of the measurement campaign, as well as the result of the modeling algorithm application are shown. Finally, some conclusions are drawn from the previous study and tests.

2 Modeling methodology

This section presents the modeling process presented in this study. To model the UE perception of a commercial network behavior, a three-step approach is proposed as follows (Fig. 2):

Fig. 2

Modeling process steps

• Measurement campaign. This first phase consists in taking measurements in commercial networks behavior as perceived by the UEs. In this sense, the values of some user performance indicators such as user download throughput has been collected.

• Analysis results and classification. By inspecting the resulting measurements, these are grouped, resulting in several scenarios that represent the behavior of the studied networks.

• Modeling. The objective of this task is to obtain a set of configuration parameters, such as the eNodeB transmission power or the bandwidth, to be applied to the testbed network in order to obtain similar user performance indicators that those obtained from measurement campaign for different considered scenarios.

In the next subsections, an overview of the measurement campaign motivation and methodology as well as the main characteristics of the algorithm used to perform the modeling is provided.

2.1 Measurement campaign

For a model to be built, a number of observations of the entity to be modeled are needed. These observations will allow extracting the underlying relations among the involved variables as well as providing a general view of the entity behavior and neglecting punctual deviations. In the context of cellular networks, a model may be extracted from its performance indicators, which take different shapes and quantify different aspects of the same system. In particular, performance information is usually gathered from network elements, such as the base stations, as counters, which count the number of times an event occurred, or key performance indicators (KPIs), which are a type of performance measurement computed from the first. Performance information may also be gathered from UEs, either by directly collecting data from radio channel measurements (such as reference signal received power, RSRP; or reference signal received quality, RSRQ) or after a complex event processing (CEP) phase is applied on signaling messages between the UE and the network.

Counters and KPIs are often collected as aggregate magnitudes (not carrying information about specific UEs) within a certain time period; typically, with a time resolution in the order of several minutes. They are periodically stored in databases belonging to the operations support systems (OSS), from which they can be retrieved for management tasks. These indicators quantify the performance of a network segment, either the radio access network (RAN) or the core network (CN), and as a result, do not allow MNOs to obtain an E2E vision of the network.

On the other hand, performance information gathered from the UEs is registered with a much higher resolution; in the order of seconds. This information may be retrieved either from the UEs themselves, by means of driving tests with UEs modified ad hoc, or from network elements. This source of performance information provides an E2E vision, allowing MNOs to follow user-centric approaches for management purposes, which is a trend nowadays [10, 11]. This additional information, which is crucial for novel network management, comes at the expense of increased processing and storing needs for MNOs. This is the type of information used to perform the modeling tasks in this paper.

2.2 Taguchi method

TM is one of the most widely used tools in the engineering experiment design scope. TM computes the values that the inputs (called factors or parameters) of a system should take in order to obtain a specific value on a target response metric. In the mobile network modeling case, these factors will be network configuration parameters, and the response metric, the UE’s metric selected to perform the modeling. To this end, the method executes successive iterations until a specified stop criterion is met. In each iteration, a set of experiments with different combinations of input factors or parameter values are performed. Each experiment consists on the use of a service by a UE (e.g., file download, video streaming). Experiment results are metrics that can represent the performance of the networks in different levels, such as radio, KPI, and key quality indicators (KQI). These metric values depend on the type of service that is being used by the user as well as the service traffic profile. Therefore, modeling results in terms of configuration parameters will depend on the service used to perform the experiments.

For each factor, TM defines a set of levels (possible values of the factors to be tested in each iteration) and an initial range of values (maximum and minimum values that could be adopted during the process).

In order to decide which combination of input factors values is selected in each experiment, TM proposes the use of an orthogonal array (OA) [12]. Thus, given a response metric and with the aim of reducing the number of experiments needed to carry out the process, an OA represents the configuration of input factors that should be used in each experiment. TM can be used with three objectives: (a) to minimize a response metric value: the smaller-the-better (STB); (b) to maximize a response metric value: the larger-the-better (LTB); and (c) to achieve a target value of a response metric: the nominal-the-best (NTB). Since the modeling purpose is to replicate the behavior of a system, the TM approach that better fits the modeling problem is the NTB.

To obtain the optimal values of the factors, TM uses a metric called signal to noise (SN). The SN metric is computed after each iteration for each factor level. The optimal level of a factor is the one that produces the highest value of the SN metric. Depending on the objective of the method: STB, LTB, and NTB, the SN metric will be calculated differently. In the NTB case, the SN metric is computed as follows:

$$ \text{SN} = -10 \cdot \log_{10} \left(\frac{\mu^{2}}{\sigma^{2}} \right) $$

(1)

where μ is the response metric mean of the experiments performed with a certain factor level and σ is their standard deviation.

Figure 3 shows the steps to follow during the process. The first step is the selection of an appropriate OA and an objective SN metric that fits the problem. Then, a mapping between the levels of each factor and the real values of these are made. To this end, it is necessary to define the maximum and minimum values to be tested for each factor. Afterwards, the experiments are carried out and, through the calculation of the SN metric, the combination of factors that provide the maximum response metric result is sought. Once the optimum values have been obtained in the iteration, the stop criterion, defined as the difference between two adjacent values of a factor, is checked. If the stop criterion is met, the process is finished; in the opposite case, the maximum and minimum values to be tested of each factor are reduced, taking as central value the optimal value in the previous iteration. Then, the mapping between the levels of each factor and the measurements of these, as well as the experiments, is made again.

Fig. 3

Operation diagram of Taguchi method

One limitation of the TM is that only one metric can be used as a response metric. However, in some cases, such as the one presented in this work, the objective of the process (maximize, minimize, or achieve a target value) is defined as a combination of different metrics. Such a situation is called the multi-response problem in TM. An approach to address this limitation is proposed in [13] and is the one followed in this paper. Specifically, the proposed solution consists of four steps: first, the SN metric is calculated for each response metric and for each combination of factors. Then, the average SN for each response metric and for each factor level and a certain weight is given at each level in relation to the maximum SN calculated for that factor and response. Next, the average weight for each factor level is calculated, considering the weight obtained in all the responses. Finally, for each factor, the factor level with the highest average weight is taken as the optimum factor level.

3 Proof of concept

In this section, first an overview of the main characteristics of the testbed used to perform the modeling tasks is provided. Then, the results obtained from the three steps that make up the modeling process, explained in the previous section, are presented.

3.1 Testbed description

The University of Malaga Heterogeneous Network (UMAHetNet) [14] is a full indoor long-term evolution (LTE) network deployed at the Telecommunication Engineering School. The RAN consists of 12 base stations in the shape of picocells. The evolved packet core (EPC) is integrated in a single compact equipment, namely, the evolved Core Network Solution (eCNS), which is composed of all core network elements (home subscriber server, HSS; mobility management entity, MME; serving gateway, S-GW; packet data network gateway, P-GW; and policy and charging rules function, PCRF), as shown in Fig. 4. On top of this, UMAHetNet is monitored and coordinated by a network management system (NMS), the Huawei solution iManager U2000.

Fig. 4

UMAHetNet general scheme

These network elements are interconnected by means of a 24-port Gbit switch providing all the equipment access to the Internet.

Both the picocells and the eCNS are fully configurable. The picocells can be managed either directly through a local management terminal (LMT) client or by means of the U2000 and eCNS clients. These two latter allow the user to fully customize all the network parameters, as well as to monitor the network status using built-in and user-defined KPIs. Furthermore, this testbed enables the user to integrate his own self-organizing network (SON) applications and algorithms, from self-configuration techniques to any self-optimizing and self-healing application.

In order to enable the integration of new management mechanisms into the NMS, as well as to allow other researchers deploy their own management functionalities in the testbed, a web service application programming interface (API) has been developed. Among the different types of web service APIs that exist (simple object access protocol, SOAP, representational state transfer, REST, common object request broker architecture, CORBA, etc.), it has been decided to implement an API of type REST. Whereas SOAP, CORBA, and other remote procedure calls (RPC) aim at defining a communication protocol between the client and the web server [15], REST is a web architecture style that uses the hypertext transfer protocol (HTTP) in the communication between these two parts. This makes the use of RPC systems more complex and inflexible. Moreover, REST allows the transmission of a greater variety of data formats (e.g. plain text, JavaScript object notation, JSON or extensible markup language, XML).

Thus, on the one hand, users communicate with the REST API using a RESTful interface. This interface is accessible using standard hypertext transfer protocol (HTTP) libraries, being easily integrable with algorithms implemented on any programming language for SON functions, monitoring, etc. On the other hand, the REST API will traduce users’ HTTP queries in (a) man-machine language (MML) commands that allow users to perform both RAN and CN configuration parameters changes and (b) structured query language (SQL) queries that allow users to consult network configuration parameters as well as performance indicators. In Fig. 5, a diagram of the REST API is shown.

Fig. 5

REST API diagram

In UMAHetNet, UEs are Linux-based devices, which are connected to the UMA picocells and which gather user’s measured performance indicators. Specifically, UEs are MONROE nodes, which are further described in [16]. The MONROE project aims at establishing a collaboration framework for European researchers in the field of mobile broadband (MBB) communications. To that end, the project allows each partner to execute MBB-related experiments in another partner’s facilities, so that information from several cellular commercial networks can be gathered and compared [17]. The experiments as well as the data collection are performed by MONROE nodes, which are permanently connected to the partner’s facilities. To manage the execution of experiments, a platform has been developed in the scope of this project [18].

Figure 6 shows the hardware structure used to perform the modeling in the UMAHetNet and the connections between its elements. This structure consists in a management device, a MONROE node and the UMAHetNet itself. The intelligent of the system is centralized in the management device (a computer, in this case). On the one hand, this element communicates with the MONROE node through a secure shell (SSH) connection. The management device sends commands to the MONROE node to perform a certain experiment and to retrieve the resulting performance measurements collected by the node during the experiments. The results of the experiments are processed by the management device, which will decide the adjustment to be made in the network configuration parameters. On the other hand, the management device communicates with the UMAHetNet through the REST API described above. The management device send to the REST API the new values of the configuration parameters, and the REST API transforms the request of the management device in MML commands and sends them to the UMAHetNet to make the configuration changes.

Fig. 6

Integration of hardware extension

3.2 Results

In this subsection, the results of the different tasks of the modeling process are shown, which are the measurement campaign, the subsequent data analysis and classification, and the use of the modeling algorithm. To carry out the experiments related to the measurement campaign and the modeling steps, a file transfer service has been selected. File transfer service is the base of most of the services demanded by UEs, especially in the MBB service category, which carries most of the cellular traffic nowadays [19] and whose most relevant performance indicator is throughput.

3.2.1 Measurement campaign

In order to validate the model of the network performance, a measurement campaign has been devised. The measurement campaign aims at obtaining a set of measurements of the UE perception of the behavior of commercial networks. These measurements will be processed and taken as the target value of the modeling algorithm. To have a broader view of the commercial networks’ behavior, different networks of some of the main European MNOs (Yoigo, Vodafone, Orange, Telia, Telenor, and TIM) have been studied. As said before, the service used for the campaign is file transfer service. By means of a battery of tests in which files of different sizes have been proved (100MB, 50MB, 10MB, 1MB, and 0.5MB), it has been determined that the optimal file size for the measurements is 1MB, achieving a tradeoff between the time to perform a complete download and the statistical significance of the results.

The measurement campaign has been performed using the MONROE platform (see Section 3.1) to manage the experiments and download the results and the MONROE nodes connected to MNOs networks to execute the experiments and gather UE metrics. The UE metrics used in this paper are collected by the nodes in two different ways. On the one hand, the nodes have installed the cURL tool [20] that provides metrics such as the setup time, the total download time, and the average achieved throughput. On the other hand, the metadata of the nodes, related to the radio signal, are collected during the execution of the experiments by means of the LTE modem available at MONROE nodes: a ZeroMQ socket. These data are the RSRP, the RSRQ, and the received signal strength indicator (RSSI).

3.2.2 Data analysis and classification of scenarios

The metrics chosen to model an external commercial network are the RSRP (in dBm) and the file transfer protocol (FTP) user throughput, averaged along the duration of each experiment. Figure 7 shows these metrics with a scatter plot, where two well differentiated behaviors can be seen. The left group of samples stands for low values of RSRP, and consequently, low throughput values. These samples may have been gathered from nodes experiencing low-signal quality, possibly due to high propagation losses or interferences. The right group, however, presents higher values of RSRP and throughput. Thus, two behaviors may be distinguished, each one representing one of the groups of samples:

• 1. Low-performance behavior. According to the samples gathered in the measurement campaign, this scenario may be characterized by an average RSRP of – 102 dBm and an average throughput of 2 Mbps.

  Fig. 7

  

  Classification of measurement campaign results. Centroids representing the average behavior of each cluster are represented with a red cross (low-performance behavior) and a black star (high-performance behavior)

• 2. High-performance behavior. This behavior may be characterized by an average RSRP of – 87 dBm and an average throughput of 5 Mbps.

3.2.3 Modeling algorithm application

The modeling algorithm application step have been carried out in the UMAHetNet network. The file size used in the file transfer service is 1 MB as in the measurement campaign. The scenario in which the modeling have been performed consists of two terminals: a MONROE node and a smartphone, each one connected to a different picocell. The picocell in which the MONROE node is connected is considered as the serving cell and the other one as the interfering cell. As said before, in this case, the metrics selected to model an external commercial network behavior are the RSRP and the UE throughput. These measurements are provided by the MONROE node as the results of the experiments. The configuration parameters that will be modified to achieve the target behavior (the input factors of TM algorithm) are the transmission power of the serving and the interfering cell, and the serving cell available bandwidth. For each configuration parameter, three levels are defined. The OA used to perform the modeling, which depends on the number of factors as well as the levels of each factors, can be found in [21]. Finally, the TM stop criterion is set to 0.2 since with this value a tradeoff between modeling accuracy and processing time is achieved. The main configuration parameters of the UMAHetNet are summarized in Table 1.

Table 1 Configuration of the UMAHetNet

Due to the fact that most of the samples obtained in the measurement campaign belong to the second of the scenarios identified in Section 3.2.2, this scenario has been selected for modeling. Thus, the target values of throughput and RSRP are 5 Mbps and – 87 dBm, respectively.

Figures 8, 9, 10, 11, and 12 show the results of the network modeling. In Figs. 8 to 10, the convergence of the configuration parameters (serving cell transmission power, interfering cell transmission power, and serving cell available bandwidth, respectively) is depicted. In these figures, it can be seen how, in each iteration, the TM controller is reducing the search range, focusing on those values of the configuration parameters for which the throughput and RSRP obtained in the simulations are closer to the target values. Finally, the values of configuration parameters, for which the target values of throughput and RSRP are achieved are in the serving cell transmission power case, – 7.6 dBm; in the interfering cell transmission power case – 14.9 dBm, and in the serving cell, the available bandwidth is 100 physical resource blocks (PRBs) (20 MHz).

Fig. 8

Serving cell transmission power adjustment

Fig. 9

Interfering cell transmission power adjustment

Fig. 10

Serving cell bandwidth adjustment

Fig. 11

FTP throughput

Fig. 12

RSRP

Figures 11 and 12 show the resulting values of throughput and RSRP obtained in each experiment (represented in blue) as well as the target values of these metrics (represented in red). It can be observed how the values of throughput and RSRP resulting from the successive experiments get closer to the throughput and RSRP target values along the iterations. So, as a result of the modeling process, specific values for the different considered configuration parameters, with which the target values of the measured user performance indicators are achieved, are obtained.

On the one hand, in Table 2, the results of the modeling process proposed in this paper are shown (testbed configuration parameters). As it is explained before, these values are the ones that should be use in the testbed network in order to achieve the target values of user performance indicators. On the other hand, the final values obtained during the modeling process for the user performance indicators: RSRP and throughput and the target values of these indicators are also listed in Table 2. Results obtained for the modeling process has been computed as the mean values of the indicators once the TM has converged. Regarding these results, it can be appreciated in a quantitative manner, that the final user performance indicators values are very close to the target ones. The final values of RSRP and throughput achieved in the modeling process are – 87.62 dBm and 5.38 Mbps, respectively, while the target values are – 87 dBm and 5 Mbps.

Table 2 Summary of the target values and the results obtained

4 Conclusion

New management network algorithms developed by researchers are rarely tested on commercial networks due to the reluctance of most MNOs. To deal with this problem, in this paper, the modeling of the UE perception of a commercial network performance using a testbed network is proposed. Firstly, a measurement campaign has been developed. To this end, several user performance indicators have been measured in different types of scenarios in commercial networks of several European operators. Secondly, by analyzing the results of the measurement campaign, it has been possible to differentiate two groups that represent the behavior of the studied networks. Finally, taking one of the previous groups as input, the modeling over a testbed network has been carried out using a TM controller. This method can be used by operators to have a preliminary assessment of the effects of a certain optimization action over their network, without actually interrupting or affecting services. It can also be used by researchers to draw better conclusions on how their algorithms will behave in a commercial network; as well as debug them for different scenarios.

As future work, the development of a more complex commercial network model is proposed. In particular, a wider range of user performance indicators will be considered, so that the different services could be better represented. New scenarios will also be considered to allow the modeling of wider range of behaviors that can be found in a commercial network with UMAHetNet. Finally, the use of new configuration parameters will be considered, to be able to model the more complex scenarios discussed above.