Modeling data-driven sensor with a novel deep echo state network

Highlights

• The proposed ADESN is capable of preserving much more input features than a traditional ESN under the same reservoir size.

• The ADESN can realize a selective memory.

• The ADESN is prominent in modeling long-term dependent tasks.

Abstract

Data-driven approach has been widely utilized in modeling soft sensor for predicting key quality variables in process engineering area. The soft sensor is generally a time dependent dynamical model between the input and the output. Echo state network (ESN) is a typical data-driven modeling tool, which has exhibited excellent performance in temporal data processing area. However, the memory mode in the traditional ESN lacks flexibility. It is sometimes hard to preserve sufficient input features in the states, especially for modeling long-term dependent soft sensors. To solve this problem, this paper proposes an asynchronously deep echo state network (ADESN), which is composed of a number of sub-reservoirs that are connected one by one in sequence. Additionally, time delay modules are inserted between every two adjacent layers. The ADESN scheme preserves more input history in the states. Moreover, it can realize a selective memory. The validity of the ADESN is demonstrated on modeling a number of numerical and real-life soft sensors.

Introduction

Soft sensor aims at building a model to estimate quality variables that are usually difficult to measure by hardware sensors. Nowadays, soft sensor has gained much attention in the process engineering area [[1], [2], [3], [4]]. There are two major techniques to build a soft sensor model: model-driven and data-driven [1,5]. Model-driven technique requires in-depth process knowledge about the system. Therefore, it is often impractical due to the complexity of industrial processes. In this respect, data-driven technique has gained increasing focus in modeling soft sensors, which are easy to develop and embedded in automatic control systems [1,[6], [7], [8]]. To date, various data-driven soft sensors have been applied, such as support vector machine (SVM) [6], Gaussian mixture model (GMM) [9], and artificial neural network (ANN) [[10], [11], [12]] etc. Among these methods, ANN is the most excellent model because of its appealing ability in handling nonlinearity and learning ability.

Soft sensor in process engineering area is usually considered as a time-dependent dynamic model. It means the output of soft sensor should depend significantly on the input history. A common way to realize this dynamical model by a traditional ANN is to make a finite input history available for the model by a sliding window technique and the current input is tapped from a delay line [13]. This way essentially transfers a dynamical mapping into a static one. Recurrent neural network (RNN) offers an alternative solution to the memory demands. RNN is a dynamical system with a high-dimensional internal state. When driven by the external input, the internal state preserves some information about the input history. It is therefore unnecessary to feed delayed input versions into the network. Hence, an RNN is more suitable to model the time dependent soft sensors than a feed-forward ANN in principle.

Echo state network (ESN) is a typical RNN with a fixed state transition structure (the reservoir) and an adaptable readout from the state space [14,15]. The neurons in the reservoir are sparsely randomly connected among each other or themselves. A significant advantage of the ESN approach is that only the output connection weights need to be adapted by learning. Algorithms for linear regression can compute the optimal output weights [15]. Thus, it significantly reduces the training complexity of RNN [16]. Theoretical study shows that the ESN can approximate any dynamical system with arbitrary accuracy [17]. Now, ESN and its large amounts of variants have been successfully applied in the modeling and control fields [[18], [19], [20], [21]].

To model a time dependent soft sensor by an ESN accurately, the ESN’s states should preserve sufficient input history. Otherwise the output layer cannot extract the required input features from the states. Theoretically, an ESN has an infinite memory capacity, but practically, the preserved features in the ESN’s states are faded overtime. It implies that the input features near the current time are significant, while the input features far away from the current time are too faint to be recalled. Many researchers have proposed a number of ways to make the ESN preserve more input features [[22], [23], [24], [25], [26]]. Most of them consider that the serious couplings among reservoir neurons inhabits the ESN’s memory capacity. It is difficult for a reservoir to generate rich dynamics because of the coupling, and further harder to preserve a large number of features implied in the input stream. In order to weaken the couplings among reservoir neurons, several improved ESN versions are presented such as decoupled ESN [22], grouped ESN [23], tree ESN [24], multiple layered ESN [25], and deep ESN (DESN) [26] etc. All these ESN’s improvements aim at reducing the couplings among reservoir neurons by dividing the single reservoir into a number of sub-reservoirs. Among these improved versions, the DESN is promising because of its deep topology and hierarchical processing of temporal information that has a remarkable biological plausibility as evidenced by studies in the field of neuroscience [27,28]. In DESN, the sub-reservoirs are connected one by one in sequence. Some experiments indicate that the DESN can generate more dynamics and has larger memory capacity than the traditional ESN [27]. From a psychological view, the memory in human brain is not continuously faded over time but tends to be selective [[29], [30], [31]]. When the audience accepts and deals with the contents of communication, they do not accept them all without analysis. They actively and selectively memorize those parts that are consistent with their own inherent concepts, interests and hobbies, and exclude the rest from their memories, so as to meet their own needs and achieve psychological balance. The selective memory mode makes the human brain capture easily the key information with limited resources. From a practical point of view, it is unnecessary and unrealistic to remember all happened things in chronological order. In most of real-life cases, remembering some key points about the happened things is enough to outline the whole picture of the event. This is also true for soft sensor modeling problems. In most soft sensor modeling cases, only a number of discrete inputs at different time steps are fed into the models [1,2,5,6]. Therefore, it is intelligent for the ESN to preserve selectively the required input information for a given task. To improve the performance of soft sensors, this paper proposes an asynchronously deep echo state network (ADESN). The ADESN is an extension of the DESN. Compared to the DESN, the ADESN meets the selective memory requirement for soft sensors by introducing the delayed links between every two adjacent layers. This selective memory mode is further helpful for the reservoir to store more required input features.

The remainder of this paper is organized as follows. In section 2, the principle of the ADESN is introduced; in section 3, the performance of ADESN is examined through modeling a number of numerical and real-life soft sensors; finally, some conclusions are given in section 4.