We introduce efficient online learning algorithms based on the Long Short Term Memory (LSTM) networks that employ the covariance information. In particular, we introduce the covariance of the present and one-time step past input vectors into the gating structure of the LSTM networks. Additionally, we include the covariance of the output vector, and we learn their weight matrices to improve the learning performance of the LSTM networks where we also provide their updates. We reduce the number of system parameters through the weight matrix factorization where we convert the LSTM weight matrices into two smaller matrices in order to achieve high learning performance with low computational complexity. Moreover, we apply the introduced approach to the Gated Recurrent Unit (GRU) architecture. In our experiments, we illustrate significant performance improvements achieved by our methods on real-life datasets with respect to the vanilla LSTM and vanilla GRU networks.

我们介绍了基于长期短期记忆（LSTM）网络的有效在线学习算法，该算法采用了协方差信息。特别地，我们将当前和过去一步输入向量的协方差引入LSTM网络的门控结构。此外，我们包括输出向量的协方差，我们学习它们的权重矩阵以改善LSTM网络的学习性能，并在其中提供它们的更新。我们通过权重矩阵分解来减少系统参数的数量，其中将LSTM权重矩阵转换为两个较小的矩阵，从而以较低的计算复杂度实现较高的学习性能。此外，我们将引入的方法应用于门控循环单元（GRU）体系结构。在我们的实验中