Abstract Long short-term memory (LSTM) is a type of recurrent neural networks that is efficient for encoding spatio-temporal features in dynamic sequences. Recent work has shown that the LSTM retains information related to the mode of variation in the input dynamic sequence which reduces the discriminability of the encoded features. To encode features robust to unseen modes of variation, we devise an LSTM adaptation named attentive mode variational LSTM. The proposed attentive mode variational LSTM utilizes the concept of attention to separate the input dynamic sequence into two parts: (1) task-relevant dynamic sequence features and (2) task-irrelevant static sequence features. The task-relevant dynamic features are used to encode and emphasize the dynamics in the input sequence. The task-irrelevant static sequence features are utilized to encode the mode of variation in the input dynamic sequence. Finally, the attentive mode variational LSTM suppresses the effect of mode variation with a shared output gate and results in a spatio-temporal feature robust to unseen variations. The effectiveness of the proposed attentive mode variational LSTM has been verified using two tasks: facial expression recognition and human action recognition. Comprehensive and extensive experiments have verified that the proposed method encodes spatio-temporal features robust to variations unseen during the training.

中文翻译：

---

编码特征对具有注意力的长短期记忆的不可见变化模式具有鲁棒性

摘要 长短期记忆 (LSTM) 是一种循环神经网络，可有效编码动态序列中的时空特征。最近的工作表明，LSTM 保留了与输入动态序列中变化模式相关的信息，这降低了编码特征的可辨别性。为了编码对看不见的变化模式具有鲁棒性的特征，我们设计了一种名为注意力模式变分 LSTM 的 LSTM 自适应。所提出的注意力模式变分 LSTM 利用注意力的概念将输入动态序列分为两部分：（1）与任务相关的动态序列特征和（2）与任务无关的静态序列特征。任务相关的动态特征用于编码和强调输入序列中的动态。利用与任务无关的静态序列特征来编码输入动态序列中的变化模式。最后，注意力模式变分 LSTM 通过共享输出门抑制模式变化的影响，并产生对看不见的变化具有鲁棒性的时空特征。已使用两项任务验证了所提出的注意力模式变分 LSTM 的有效性：面部表情识别和人体动作识别。全面而广泛的实验已经证实，所提出的方法编码的时空特征对训练期间看不见的变化具有鲁棒性。已使用两项任务验证了所提出的注意力模式变分 LSTM 的有效性：面部表情识别和人体动作识别。全面而广泛的实验已经证实，所提出的方法编码的时空特征对训练期间看不见的变化具有鲁棒性。已使用两项任务验证了所提出的注意力模式变分 LSTM 的有效性：面部表情识别和人体动作识别。全面而广泛的实验已经证实，所提出的方法编码的时空特征对训练期间看不见的变化具有鲁棒性。