An efficient Long Short-Term Memory model based on Laplacian Eigenmap in artificial neural networks

Abstract

A new algorithm for data prediction based on the Laplacian Eigenmap (LE) is presented. We construct the Long Short-Term Memory model with the application of the LE in artificial neural networks. The new Long Short-Term Memory model based on Laplacian Eigenmap (LE-LSTM) reserves the characteristics of original data using the eigenvectors derived from the Laplacian matrix of the data matrix. LE-LSTM introduces the projection layer embedding data into a lower dimension space so that it improves the efficiency. With the implementation of LE, LE-LSTM provides higher accuracy and less running time on various simulated data sets with characteristics of multivariate, sequential, and time-series. In comparison with previously reported algorithms such as stochastic gradient descent and artificial neural network with three layers, LE-LSTM leads to many more successful runs and learns much faster. The algorithm provides a computationally efficient approach to most of the artificial neural network data sets.

Introduction

Artificial Neural Network (ANN) has attracted an enormous amount of attention in recent years in the artificial intelligence area [1]. Researchers have concentrated on the ANN applications including image processing [2], [3], speech recognition [4], medical diagnosis [5], [6], robots [7], [8], [9], semantic object parsing [10], speech enhancement [11], stock price prediction [12], etc. Comparing to the traditional prediction algorithms, the ANN can fit data with arbitrary shapes and get higher prediction accuracy. However, the ANN has high computationalcost with the increasing numbers of the hidden layers or neurons [13]. Therefore, it is still a challenging area to find a highly accurate and efficient ANN algorithm for various data sets with different features.

A general ANN is constituted by three layers: input layer, hidden layer, and output layer. There is no feedback mechanism when ANN processes the sequence information, which leads to the decline of prediction accuracy. As an in-depth study on neural network, the researchers found that this defect of traditional neural network restricts its long-term development. Recurrent neural network (RNN) [15], a correction to this defect, introduces a feedback mechanism in the hidden layer, which can study the contextual information of sequence data. Recently, RNN has been applied into various research fields, such as sentiment analysis [16], machine translation [17], etc. RNN trains the data using backpropagation through time algorithm [18], therefore, there still exists the defect of high computational complexity. In order to satisfy the real-time requirement of special scenarios, improve the prediction accuracy, reduce the loss, and speed up the convergence, many researchers presented various improved methods to optimize the RNN algorithms. Mohammed et al. [19] formulated the kinematic control problem of Stewart platforms to constrain quadratic programming, and obtained the Karush–Kuhn–Tucker conditions of the problem by considering the dual space, and then, designed a dynamic neural network to solve the optimization problem recurrently. A novel primal–dual neural network was designed by Li et al. [20], this model directly took noise control into account and was able to reach the accurate tracking of reference trajectories in a noisy environment. Xiao et al. proposed two nonlinear recurrent neural networks based on two nonlinear activation functions to solve general time-varying linear matrix equations [21]. Kusupati et al. developed the FastRNN and FastGRNN algorithms to address the twin RNN limitations of inaccurate training and inefficient prediction [22]. Xiao et al. presented a systematic and constructive procedure using zeroing neural dynamics to design control laws based on the efficient solution of dynamic Lyapunov equation [23], and then designed two robust nonlinear zeroing neural networks by adding two novel nonlinear activation functions for finding the solution of the Lyapunov equation in the presence of various noises [24]. Miikkulainen et al. proposed an automated method, CoDeepNEAT, for optimizing deep learning architectures through evolution [25].

Based on RNN, a mainstream deep learning model, Long Short-Term Memory (LSTM) algorithm [26] adds the step of filtering for previous states, which can select the states with the greater influence to the current states. Therefore, LSTM can solve the vanishing gradient problem, which can be explained that the gradient value will become very small and lead to stop learning as time going on. Kuchaiev et al. presented two simple ways of reducing the number of parameters and accelerating the training of large LSTM networks [27]. Karim et al. proposed the augmentation of fully convolutional networks with LSTM and RNN sub-modules for time series classification [28]. Elsaid et al. developed a RNN capable of predicting aircraft engine vibrations using LSTM neurons [29]. Miikkulainen et al. proposed a new method, evolution of a tree-based encoding of the gated memory nodes [30].

Using a traditional approach, the LSTM algorithm usually maps the original data into memory unit according to the shape of data, which will lead to a situation: the mapped data may lose the features or hidden information in original data. Essentially, this defect may result in the inaccuracy of the prediction model. In order to fix this defect, we introduced the idea of Laplacian Eigenmap (LE) [31] to solve the problem of data encapsulation, and proposed a new algorithm Long Short-Term Memory based on Laplacian Eigenmap (LE-LSTM). This algorithm is similar to the standard RNN [15] with a certain number of hidden layers. We replaced each normal node as a memory unit containing one node with a self-connecting loop edge and the fixed weight $1$, which indicates the feedback with a decay of $1$ time step [26]. This approach uses the LE to transform the original data without losing information hidden in data and can acquire good performance in prediction.

The remainder of this paper is organized as follows. In Section 2, we describe the idea, model, steps, computational complexity, and pseudocode of the LE-LSTM algorithm. We present some evaluation metrics and discuss the numerical results in Section 3. In Section 4, experimental results are discussed and we show that the proposed algorithm has higher training, testing accuracy and lower training, testing loss. The running time is also less than the LSTM. Furthermore, the details of the numerical performance of LE-LSTM and the other competitive algorithms are discussed. We draw a conclusion and give the future works in Section 5.

Overall our paper makes the following contributions:

• This study presents a new method in data prediction via Long Short-Term Memory Model based on Laplacian Eigenmap.

• The LE-LSTM reserves the characteristics of original data using eigenvectors derived from the Laplacian matrix of the data matrix.

• This algorithm provides higher accuracy and less running time on representative data sets with characteristics of multivariate, sequential, and time-series than other state-of-the-art algorithms.

• This proposed algorithm yields satisfactory performance and computational efficiency.