Skip to main content
SpringerLink
Log in

What went wrong? Identification of everyday object manipulation anomalies

  • Original Research Paper
  • Published:
Intelligent Service Robotics Aims and scope Submit manuscript

Abstract

Enhancing the abilities of service robots is important for expanding what they can achieve in everyday manipulation tasks. In addition, it is also essential to ensure that they can determine what they cannot achieve. Such necessity may arise due to anomalies during task execution. These situations should be detected and identified to overcome and recover from them. Identification necessitates a deeper time series analysis of onboard sensor readings to keep track of and relate anomaly indicators since some indicators may be perceived long before the detection of an anomaly. These sensor readings are usually taken asynchronously and need to be fused effectively for correct interpretations. In this paper, we propose a multimodal long short-term memory-based (LSTM-based) anomaly identification approach that takes into account real-time observations by fusing visual, auditory and proprioceptive sensory modalities during everyday object manipulation tasks. The symptoms of anomalies are first trained and then are classified based on the learned models in real time. We first provide a comparative analysis of our method with hidden Markov models (HMMs), conditional random fields (CRFs) and gated recurring units (GRUs) on a Baxter robot executing everyday object manipulation scenarios. Then, we analyze the impact of each modality and various feature extraction techniques on the performance of the identification problem. We show that our method has the ability to identify anomalies by capturing long-term dependencies between the anomaly indicators. The results indicate that the LSTM-based anomaly identification method outperforms the closest baseline with a 2% improvement of f-score (0.92) in classifying anomalies that occur during run-time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. http://www.chokkan.org/software/crfsuite/.

References

  1. Abid A, Khan MT, de Silva C (2015) Fault detection in mobile robots using sensor fusion. In: 2015 10th international conference on computer science & education (ICCSE), IEEE, pp 8–13

  2. Altan D, Sariel S (2016) Empirical analysis of probabilistic methods for failure isolation in robots. In: Proceedings of the IEEE/RSJ IEEE/RSJ international conference on intelligent robots and systems (IROS) Workshop Cognit. Robot. (CogRob)

  3. Altan D, Sariel-Talay S (2014) Probabilistic failure isolation for cognitive robots. In: Proceedings of the Florida artificial intelligence research society conference (FLAIRS)

  4. Azzalini D, Castellini A, Luperto M, Farinelli A, Amigoni F (2020) HMMs for anomaly detection in autonomous robots. In: Proceedings of the 19th international conference on autonomous agents and multiagent systems, pp 105–113

  5. Baghernezhad F, Khorasani K (2016) Computationally intelligent strategies for robust fault detection, isolation, and identification of mobile robots. Neurocomputing 171:335–346

    Article  Google Scholar 

  6. Baum LE, Petrie T (1966) Statistical inference for probabilistic functions of finite state Markov chains. Ann Math Stat 37(6):1554–1563

    Article  MathSciNet  Google Scholar 

  7. Bowkett J, Burdick J, Matthies L, Detry R (2018) Semantic understanding of task outcomes: Visually identifying failure modes autonomously discovered in simulation. In: Representing a complex world: perception, inference, and learning for joint semantic, geometric, and physical understanding (ICRA 2018 Workshop)

  8. Carlson J, Murphy RR, Nelson A (2004) Follow-up analysis of mobile robot failures. In: 2004 IEEE International Conference on Robotics and automation, 2004. Proceedings. ICRA’04, IEEE, vol 5, pp. 4987–4994

  9. Chandola V, Banerjee A, Kumar V (2009) Anomaly detection: a survey. ACM Comput Surv 41(3):1–58

    Article  Google Scholar 

  10. Chang Chein-I, Chiang Shao-Shan (2002) Anomaly detection and classification for hyperspectral imagery. IEEE Trans Geosci Remote Sens 40(6):1314–1325

    Article  Google Scholar 

  11. Crammer K, Kulesza A, Dredze M (2009) Adaptive regularization of weight vectors. In: Advances in neural information processing systems, pp 414–422

  12. Di Lello E, De Laet T, Bruyninckx H (2012) Hierarchical dirichlet process hidden markov models for abnormality detection in robotic assembly. In: Neural information processing systems, 2012/12/03–2012/12/08, Location: Lake Tahoe, Nevada

  13. Di Lello E, Klotzbucher M, De Laet T, Bruyninckx H (2013) Bayesian time-series models for continuous fault detection and recognition in industrial robotic tasks. In: 2013 IEEE/RSJ international conference on intelligent robots and systems (IROS), IEEE, pp 5827–5833

  14. Elmoufidi A, El Fahssi K, Jai-Andaloussi S, Sekkaki A, Gwenole Q, Lamard M (2017) Anomaly classification in digital mammography based on multiple-instance learning. IET Image Process 12(3):320–328

    Article  Google Scholar 

  15. Ersen M, Oztop E, Sariel S (2017) Cognition-enabled robot manipulation in human environments: requirements, recent work, and open problems. IEEE Robot Autom Mag 24(3):108–122. https://doi.org/10.1109/MRA.2016.2616538

    Article  Google Scholar 

  16. Ersen M, Sariel-Talay S, Yalcin H (2013) Extracting spatial relations among objects for failure detection on Visual and Spatial Cognition. p 13

  17. Forney GD (1973) The viterbi algorithm. Proc IEEE 61(3):268–278

    Article  MathSciNet  Google Scholar 

  18. Fritz C (2005) Execution monitoring—a survey. Tech. Rep, University of Toronto

  19. Göbelbecker M, Keller T, Eyerich P, Brenner M, Nebel B (2010) Coming up with good excuses: what to do when no plan can be found. In: Cognitive robotics, 10081

  20. Goel P, Dedeoglu G, Roumeliotis SI, Sukhatme GS (2000) Fault detection and identification in a mobile robot using multiple model estimation and neural network. In: Proceedings 2000 ICRA. Millennium conference. IEEE international conference on robotics and automation. Symposia proceedings (Cat. No. 00CH37065), IEEE, vol 3, pp 2302–2309

  21. Gspandl S, Podesser S, Reip M, Steinbauer G, Wolfram M (2012) A dependable perception-decision-execution cycle for autonomous robots. In: IEEE international conference on robotics and automation (ICRA), pp 2992–2998. https://doi.org/10.1109/ICRA.2012.6225078

  22. Han W, Chan CF, Choy CS, Pun KP (2006) An efficient MFCC extraction method in speech recognition. In: 2006 IEEE international symposium on circuits and systems, 2006. ISCAS 2006. Proceedings, IEEE, pp 4–pp

  23. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  24. Hinterstoisser S, Cagniart C, Ilic S, Sturm P, Navab N, Fua P, Lepetit V (2012) Gradient response maps for real-time detection of textureless objects. IEEE Trans Pattern Anal Mach Intell 34(5):876–888

    Article  Google Scholar 

  25. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  26. Inceoglu A, Ince G, Yaslan Y, Sariel S (2018) Comparative assessment of sensing modalities on manipulation failure detection. In: IEEE ICRA workshop on perception, inference and learning for joint semantic, geometric and physical understanding

  27. Inceoglu A, Ince G, Yaslan Y, Sariel S (2018) Failure detection using proprioceptive, auditory and visual modalities. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 2491–2496

  28. Inceoglu A, Koc C, Kanat BO, Ersen M, Sariel S (2018) Continuous visual world modeling for autonomous robot manipulation. IEEE Trans Syst Man Cybern Syst. https://doi.org/10.1109/TSMC.2017.2787482

    Article  Google Scholar 

  29. Isermann R (2005) Model-based fault-detection and diagnosis-status and applications. Ann Rev Control 29(1):71–85

    Article  Google Scholar 

  30. Isermann R, Ballé P (1997) Trends in the application of model-based fault detection and diagnosis of technical processes. Control Eng Pract 5(5):709–719

    Article  Google Scholar 

  31. Karapinar S, Altan D, Sariel-Talay S (2012) A robust planning framework for cognitive robots. In: Proceedings of the AAAI-12 workshop on cognitive robotics (CogRob)

  32. Karapinar S, Sariel S (2015) Cognitive robots learning failure contexts through real-world experimentation. Auton Robots 39(4):469–485. https://doi.org/10.1007/s10514-015-9471-y

    Article  Google Scholar 

  33. Khalastchi E, Kalech M (2018) On fault detection and diagnosis in robotic systems. ACM Comput Surv 51(1):1–24

    Article  Google Scholar 

  34. Khalastchi E, Kalech M, Kaminka GA, Lin R (2015) Online data-driven anomaly detection in autonomous robots. Knowl Inf Syst 43(3):657–688

    Article  Google Scholar 

  35. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. CoRR arXiv:1412.6980

  36. Krizhevsky A (2014) One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997

  37. Lafferty JD, McCallum A, Pereira FCN (2001) Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of the eighteenth international conference on machine learning, ICML ’01, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, pp 282–289

  38. Lopez F, Saez M, Shao Y, Balta EC, Moyne J, Mao ZM, Barton K, Tilbury D (2017) Categorization of anomalies in smart manufacturing systems to support the selection of detection mechanisms. IEEE Robot Autom Lett 2(4):1885–1892

    Article  Google Scholar 

  39. Luo S, Wu H, Lin H, Duan S, Guan Y, Rojas J (2018) Fast, robust, and versatile event detection through hmm belief state gradient measures. IEEE Intern, (in–print)

  40. Maxion RA (1990) Anomaly detection for diagnosis. In: 20th International Symposium on Digest of Papers. Fault-Tolerant Computing, IEEE Computer Society, pp 20–21

  41. Mendoza JP, Veloso M, Simmons R (2015) Plan execution monitoring through detection of unmet expectations about action outcomes. In: 2015 IEEE international conference on robotics and automation (ICRA), IEEE, pp 3247–3252

  42. Meng W, Liu Y, Zhang S, Pei D, Dong H, Song L, Luo X (2018) Device-agnostic log anomaly classification with partial labels. In: 2018 IEEE/ACM 26th international symposium on quality of service (IWQoS), IEEE, pp 1–6

  43. Morais MG, Meneguzzi FR, Bordini RH, Amory AM (2015) Distributed fault diagnosis for multiple mobile robots using an agent programming language. In: 2015 international conference on advanced robotics (ICAR), IEEE, pp 395–400

  44. Nan C, Khan F, Iqbal MT (2008) Real-time fault diagnosis using knowledge-based expert system. Process Saf Environ Prot 86(1):55–71

    Article  Google Scholar 

  45. Olivato M, Cotugno O, Brigato L, Bloisi D, Farinelli A, Iocchi L (2019) A comparative analysis on the use of autoencoders for robot security anomaly detection. In: 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS), IEEE, pp 984–989

  46. Park D, Erickson Z, Bhattacharjee T, Kemp CC (2016) Multimodal execution monitoring for anomaly detection during robot manipulation. In: 2016 IEEE international conference on robotics and automation (ICRA), IEEE, pp 407–414

  47. Park D, Hoshi Y, Kemp CC (2018) A multimodal anomaly detector for robot-assisted feeding using an LSTM-based variational autoencoder. IEEE Robot Autom Lett 3(3):1544–1551

    Article  Google Scholar 

  48. Park D, Kim H, Hoshi Y, Erickson Z, Kapusta A, Kemp CC (2017) A multimodal execution monitor with anomaly classification for robot-assisted feeding. In: 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), IEEE, pp 5406–5413

  49. Park D, Kim H, Kemp CC (2018) Multimodal anomaly detection for assistive robots. Auton Robots 43:1–19

    Google Scholar 

  50. Pettersson O (2005) Execution monitoring in robotics: a survey. Robot Auton Syst 53:73–88

    Article  Google Scholar 

  51. Quigley M, Conley K, Gerkey B, Faust J, Foote T, Leibs J, Wheeler R, Ng AY (2009) Ros: an open-source robot operating system. In: ICRA workshop on open source software, Kobe, Japan, vol 3, p 5

  52. Rigatos GG (2009) Particle and kalman filtering for fault diagnosis in dc motors. In: Vehicle power and propulsion conference, 2009. VPPC’09. IEEE, IEEE, pp 1228–1235

  53. Saltali I, Sariel S, Ince G (2016) Scene analysis through auditory event monitoring. In: Proceedings of the international workshop on social learning and multimodal interaction for designing artificial agents, DAA ’16, ACM, New York, NY, USA pp 5:1–5:6. https://doi.org/10.1145/3005338.3005343

  54. Saputro DRS, Widyaningsih P (2017) Limited memory broyden-fletcher-goldfarb-shanno (l-bfgs) method for the parameter estimation on geographically weighted ordinal logistic regression model (gwolr). In: AIP conference proceedings, AIP Publishing, vol 1868, p 040009

  55. Scime L, Beuth J (2018) Anomaly detection and classification in a laser powder bed additive manufacturing process using a trained computer vision algorithm. Addit Manuf 19:114–126. https://doi.org/10.1016/j.addma.2017.11.009

    Article  Google Scholar 

  56. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  57. Steinbauer G, Wotawa F (2009) Robust plan execution using model-based reasoning. Adv Robot 23(10):1315–1326

    Article  Google Scholar 

  58. Trevor AJB, Gedikli S, Rusu RB, Christensen HI (2013) Efficient organized point cloud segmentation with connected components. In: 3rd workshop on semantic perception, mapping and exploration (SPME). Karlsruhe, Germany

  59. Ullah W, Ullah A, Haq IU, Muhammad K, Sajjad M, Baik SW (2020) CNN features with bi-directional LSTM for real-time anomaly detection in surveillance networks. Multimed Tools Appl, pp 1–17

  60. Verma V, Gordon G, Simmons R, Thrun S (2004) Real-time fault diagnosis [robot fault diagnosis]. Robot Autom Mag IEEE 11(2):56–66

    Article  Google Scholar 

  61. Wellhausen L, Ranftl R, Hutter M (2020) Safe robot navigation via multi-modal anomaly detection. IEEE Robot Autom Lett 5(2):1326–1333

    Article  Google Scholar 

  62. Wu H, Guan Y, Rojas J (2019) Analysis of multimodal Bayesian nonparametric autoregressive hidden Markov models for process monitoring in robotic contact tasks. Int J Adv Robot Syst 16(2):1729881419834840

    Article  Google Scholar 

  63. Wu H, Guan Y, Rojas J (2019) A latent state-based multimodal execution monitor with anomaly detection and classification for robot introspection. Appl Sci 9(6):1072. https://doi.org/10.3390/app9061072

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Computer and Informatics Engineering, Artificial Intelligence and Robotics Laboratory, Istanbul Technical University, Istanbul, Turkey

    Dogan Altan & Sanem Sariel

Authors
  1. Dogan Altan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sanem Sariel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dogan Altan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This research was partially supported by Istanbul Technical University Scientific Research Projects (ITU-BAP) under Grant 40240 and the Scientific and Technological Research Council of Turkey (TUBITAK) under Grant 115E-368.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Altan, D., Sariel, S. What went wrong? Identification of everyday object manipulation anomalies. Intel Serv Robotics 14, 215–234 (2021). https://doi.org/10.1007/s11370-021-00355-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11370-021-00355-w

Keywords

Search

Navigation