Enhancing the abilities of service robots is important for expanding what they can achieve in everyday manipulation tasks. In addition, it is also essential to ensure that they can determine what they cannot achieve. Such necessity may arise due to anomalies during task execution. These situations should be detected and identified to overcome and recover from them. Identification necessitates a deeper time series analysis of onboard sensor readings to keep track of and relate anomaly indicators since some indicators may be perceived long before the detection of an anomaly. These sensor readings are usually taken asynchronously and need to be fused effectively for correct interpretations. In this paper, we propose a multimodal long short-term memory-based (LSTM-based) anomaly identification approach that takes into account real-time observations by fusing visual, auditory and proprioceptive sensory modalities during everyday object manipulation tasks. The symptoms of anomalies are first trained and then are classified based on the learned models in real time. We first provide a comparative analysis of our method with hidden Markov models (HMMs), conditional random fields (CRFs) and gated recurring units (GRUs) on a Baxter robot executing everyday object manipulation scenarios. Then, we analyze the impact of each modality and various feature extraction techniques on the performance of the identification problem. We show that our method has the ability to identify anomalies by capturing long-term dependencies between the anomaly indicators. The results indicate that the LSTM-based anomaly identification method outperforms the closest baseline with a 2% improvement of f-score (0.92) in classifying anomalies that occur during run-time.

增强服务机器人的能力对于扩展他们在日常操纵任务中可以实现的目标非常重要。另外，确保他们可以确定自己无法实现的目标也很重要。由于任务执行期间的异常，可能会出现这种必要性。这些情况应被发现和识别以克服并从中恢复。识别需要对车载传感器读数进行更深的时间序列分析，以跟踪和关联异常指标，因为某些指标可能在检测到异常之前就已被感知到。这些传感器读数通常是异步获取的，需要有效地融合以正确解释。在本文中，我们提出了一种基于多模式长期短期记忆（基于LSTM）的异常识别方法，该方法通过在日常对象操作任务中融合视觉，听觉和本体感受感官方式来考虑实时观察。首先对异常的症状进行训练，然后根据学习的模型进行实时分类。我们首先在执行日常对象操作场景的Baxter机器人上使用隐马尔可夫模型（HMM），条件随机场（CRF）和门控循环单元（GRU）对我们的方法进行比较分析。然后，我们分析了每种模式和各种特征提取技术对识别问题性能的影响。我们表明，我们的方法具有通过捕获异常指标之间的长期依存关系来识别异常的能力。