A multi-energy load prediction model based on deep multi-task learning and ensemble approach for regional integrated energy systems

Highlights

• CGRU can extract more efficient features and model time series dynamically.

• Different network structures are used to meet prediction needs of different loads.

• HUMTL is used to deeper dig coupling relations among various energy systems.

• GBRT realizes the sharing of prediction results learning in different degrees.

• Proposed multi-energy prediction model has advantages in accuracy and applicability.

Abstract

Regional integrated energy system (RIES) plays an important role in the energy economy because of its advantages such as low environmental pollution and high efficiency cascade energy utilization. In order to ensure the operational efficiency and reliability of RIES, the accurate prediction of energy demand has become a crucial task. To this end, this paper proposes a novel multi-energy load prediction model based on deep multi-task learning and ensemble approach for RIES. Its novelty lies in the following four aspects: (1) considering the high-dimensional temporal and spatial features, a hybrid network based on convolutional neural network (CNN) and gated recurrent unit (GRU) is utilized to extract high-dimensional abstract features and model nonlinear time series dynamically; (2) to meet the prediction requirements of various loads, three GRU networks with different structures are designed, which can adapt to different types of loads with various fluctuations; (3) considering the coupling relations, an enhanced multi-task learning with homoscedastic uncertainty (HUMTL) is proposed, which can better make the prediction tasks of various loads achieve the optimum simultaneously; (4) to realize the sharing of learning results of different structure networks, ensemble approach based on gradient boosting regressor tree (GBRT) is adopted, which can make a weighted summary by the prediction results of various energy features learning in different degrees. Numerical example shows that the proposed model can dig the coupling relations among various energy systems deeper, explore the temporal and spatial correlation of multi-energy loads further, and it has higher prediction accuracy and better prediction applicability than other current advanced models.

Introduction

With the rapid development of social economy, the coordinated development of energy utilization and environmental protection is facing enormous challenges [1]. For the construction of traditional energy systems, the planning, design and operation of various energy systems tend to separate from each other, which breaks the coupling among different types of energy systems, resulting in the failure to ensure the security, self-healing ability and energy utilization ratio of energy systems effectively [2]. Against this background, a great deal of countries turned their attentions to integrated energy system [3]. Among them, regional integrated energy system (RIES) is the basis for comprehensive analysis of wide-area energy, and it is also the specific performance characteristic of integrated energy system.

RIES is generally composed of energy generation, distributed energy sources, energy storage components, loads, etc. It has the characteristics of high efficiency, security and controllability. RIES is regarded as the main carrier mode of urban energy for human society in the next few decades. It can realize the conversion utilization, collaborative optimization and coupling complementary of various energies such as cooling, heating and power. It is essential in building a new generation of energy system. Nowadays, the energy system has a lot of problems in terms of energy utilization. For instance, in power system, coal-fired power units can only convert about 33% burning energy of fossil fuels into power energy, and the waste heating is not reused, resulting in the loss of energy [4]; in heating system, only about 80% fossil energy can be converted into heating energy, the high-temperature steam generated is not used for power generation and is wasted; the cooling system is often accompanied by a significant amount of power consumption, which makes the pressure of power loads even greater. Therefore, facing the existing drawbacks of energy utilization in the context of RIES, digging into the coupling deeply among different energy systems to achieve more flexible, efficient and accurate multi-energy load prediction has become an urgent issue today. Multi-energy load prediction can provide important data support for planning and operation design of RIES, and it has great practical significance and economic value.

Load prediction has indispensable effect on energy system, so that a lot of scholars carried out relevant researches on this issue. At present, there are great achievements for load prediction of a single type. In terms of power load prediction [5], presented a composite method based on a multi-layer perceptron (MLP) neural network, particle swarm optimization and improved ant lion optimizer to solve the power load prediction problem. Ref. [6] proposed a method of support vector regress to forecast building energy consumption, and the two parameters of support vector regress were adjusted using the cross validation with grid searching method based on radial-basis function kernel. Ref. [7] constructed a deep belief network (DBN) model made up from multiple layers of restricted Boltzmann machines for power load prediction. In terms of heating load prediction [8], constructed an expert system composed of linear regression, extremely randomized trees regression, feed-forward neural network and support vector machine to predict district heating demand. Ref. [9] proposed a generally applicable, simple and adaptive prediction method based on a linear regression model for the short-term heating load. Ref. [10] proposed a heating load prediction model based on convolutional neural network (CNN) and recurrent neural network. In terms of cooling load prediction [11], utilized the long short-term memory (LSTM) network for predicting the periodic cooling load consumption. Ref. [12] presented an air-conditioning cooling load prediction method based on weather forecast, internal occupancy density and multiple linear feedback regression model. In addition, some scholars also conducted some studies to predict loads of multiple types simultaneously. Ref. [13] considered the different effects of variables and used nonlinear autoregressive models to predict cooling, heating and power loads. Ref. [14] proposed a cooling and power loads prediction model, which combines multiple linear regression models with seasonal autoregressive moving average models. Ref. [15] used an autoregressive moving average with exogenous inputs model to predict cooling, heating and power loads. However, the above researches failed to consider the coupling relations among different types of energy systems. Refs. [16], [17] constructed power-gas and cooling-heating-power load prediction models by radial basis function neural network and LSTM respectively, and they both utilized Pearson correlation coefficient to analyze the coupling relations among different types of energy systems, but this way of digging coupling is not sufficient. Ref. [18] established a combined power-heating–cooling-gas load prediction model based on multi-task learning and least square support vector machine, but it has low computational efficiency. Nowadays, there are few efficient and accurate multi-energy load prediction models that can fully dig the coupling among various energy systems.

In recent years, deep learning has gradually become the most potential technology in machine learning. It has powerful revealing capabilities of nonlinear and complex structure in big data environment. Among them, recurrent neural network has been favored by lots of scholars. Compared with conventional machine learning methods, recurrent neural network can take the temporal correlation of time series into account, so it can describe the changes of time series more comprehensively [19]. LSTM network is a special recurrent neural network which can avoid the disadvantages of gradient vanishing and gradient explosion of recurrent neural network due to its special structure [20], but it still has the deficiency of long-time training. On the basis of LSTM, gated recurrent unit (GRU) network is proposed to make the network structure simpler than LSTM and accelerate the rate of convergence while guaranteeing the accuracy. At present, there have been some studies based on GRU to predict a single type of load such as power, photovoltaic and heating loads [21], [22], [23]. Although LSTM and GRU have many advantages in time series prediction, they do not consider the spatial feature of sequence data. And when time series is too long, LSTM and GRU cannot capture the interdependencies of short sequences [24]. In view of these problems [25], presented a novel supervised video summarization scheme based on CNN and LSTM. Ref. [26] proposed a spatially recursive model based on CNN, LSTM and deep attention to better realize fine-grained visual recognition.

Most of conventional machine learning methods only focus on single-task learning, which may result in insufficient training and under-fitting. In addition, for more complex problems, there is abundant correlation information between each task, but the ability of single-task learning is not enough to fully exploit it. In order to solve the problems of single-task learning, Caruana proposed the multi-task learning mechanism [27], so that each task can be learned in parallel to supplement each other. In summary, multi-task learning is putting multiple related tasks together to learn, so that the learning of each task can affect and improve the learning of other tasks, achieve information sharing, and finally obtain the optimal results of multiple tasks simultaneously. Ref. [28] compared single-task learning and multi-task learning based on artificial neural networks, and the research indicated that multi-task learning can improve the prediction accuracy of each task. Ref. [29] proposed an air quality prediction model based on deep neural network, DBN and multi-task learning to accomplish the three prediction tasks of PM_2.5, NO₂ and SO₂. Ref. [30] used a multiscale convolutional network and multi-task learning to address three different computer vision tasks including depth prediction, surface normal estimation and semantic labeling. As for the processing of the loss generated by each task in multi-task learning, one measure is to add all losses directly, but this method always makes it impossible for all tasks to achieve a win-win result. Another measure is to replace the sum of the losses with a naive weighted sum in multi-task learning, but this method will be cumbersome because it involves the adjustment of an additional hyperparameter. It is critical for multi-task learning to find an appropriate weighting strategy of the losses to minimize the total loss. Ref. [31] pointed out that there are two kinds of uncertainties, aleatoric uncertainty and epistemic uncertainty, in Bayesian neural network modeling. Homoscedastic uncertainty is a type of aleatoric uncertainty which does not rely on input data but can capture uncertainty of the task itself and varies between different tasks. Ref. [32] first applied homoscedastic uncertainty to loss adjustment of multi-task learning, and it proposed a method for simultaneous multi-class instance segmentation. Ref. [33] also introduced homoscedastic uncertainty into the loss, established a new capsule networks based on a fully Bayesian formulation and showed its potential by solving an ill-posed problem.

In addition, scholars tried to combine several prediction models, designed the ensemble approach of load prediction and achieved higher prediction accuracy [34]. Ref. [35] proposed a power load prediction model combining artificial neural network and extreme learning machine. Ref. [36] proposed a cooling load prediction method based on empirical mode decomposition and DBN. Ref. [37] innovatively used the same prediction model to generate multiple results by clustering and grouping, and then weighted these results to obtain the final power load prediction values. Moreover, Ref. [38] presented a novel method, gradient boosting regressor tree (GBRT), to predict the nonlinear and imbalanced incident clearance time based on different types of explanatory variables. It pointed out that GBRT can compute the relative importance of each explanatory variable that contributes to the response variable, and GBRT shows the superior prediction accuracy and interpretation ability among many artificial intelligence methods. Ref. [39] presented a precise very short-term load prediction model based on GBRT. Ref. [40] utilized GBRT to propose a multi-site solar power generation prediction approach, and the superiority of GBRT was proved.

At present, the load prediction of single energy system is relatively mature, and a lot of surprising results are obtained. In RIES, different types of loads are not only related to their own historical loads and external factors, but also have coupling relations among them. The existing prediction model of single energy load cannot be directly applied to the prediction for multiple types of loads in RIES, and there are few studies on multi-energy load prediction at present. Therefore, in order to bridge this gap and dig the coupling relations deeply, based on existing researches, this paper proposes a novel multi-energy load prediction model based on deep multi-task learning with homoscedastic uncertainty (HUMTL) and ensemble approach for RIES, named as HUMTL-CGRUG model. Firstly, CNN is utilized to extract features from the input data of model, then the extracted features are input into GRU networks which have three different structures for training. Secondly, the three trained hybrid networks (named as CGRU) are carried out HUMTL respectively, sequentially obtain the three sets of prediction results. Finally, these different network models (named as HUMTL-CGRU) are integrated by GBRT, then the three sets of prediction values are weighted and finally summed to get the cooling, heating and power prediction results. As stated above, the key contributions of this paper are as follows:

• CGRU hybrid network is utilized to extract high-dimensional temporal and spatial features, and model nonlinear time series dynamically.

• GRU networks with different structures are designed. It can adapt to different types of loads with various fluctuations and meet the prediction requirements of various loads.

• HUMTL is introduced to fully dig the coupling relations among various energy systems and better make all prediction tasks of different types of loads achieve the optimum simultaneously.

• Ensemble approach based on GBRT is adopted, which can make a weighted summary by the prediction results of various energy features learning in different degrees, and realize the sharing of learning results of different structure networks.

The remainder of this paper is organized as follows. Section 2 analyzes the characteristics of RIES. Section 3 introduces the proposed HUMTL-CGRUG model, and then elaborates the CGRU hybrid network model, the HUMTL and the ensemble approach based on GBRT respectively. Section 4 analyzes the contributions of CNN, HUMTL and ensemble approach based on GBRT to the proposed model, compares the proposed model with five contrast models, and discusses the dimension influence of input and the correlation analysis of prediction results. Section 5 draws the conclusion.