Skip to main content
SpringerLink
Log in

Humans interacting with multi-robot systems: a natural affect-based approach

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

This paper proposes a novel human–multi-robot-system interaction approach that enjoys two main features: natural interaction and affect-based adaptation of robots behavior. Specifically, the proposed system enables interaction by means of a wrist-worn device, such as a commercial smartwatch, which allows to track user’s movements and heart activity. Thus, on the one side, the proposed system allows the user to intuitively drive the robots by establishing a natural mapping between wrist movements and robots velocity. On the other side, the system estimates user’s mental fatigue during interaction by means of the analysis of heart rate variability. The proposed interaction system adapts then the behavior of the multi-robot system when the interacting user gets overwhelmed with the interaction and control task, which is then simplified. Experimental validation is provided, to show the effectiveness of the proposed system. First, the natural and affect-based interaction are considered separately. Then, the approach is tested considering a complex realistic scenario, which is simulated in virtual reality in order to get an immersive and realistic interaction experience. The results of the experimental validation clearly show that the proposed affect-based adaptive system leads to relieving the user’s fatigue and mental workload.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  • Acharya, U. R., Joseph, K. P., Kannathal, N., Lim, C. M., & Suri, J. S. (2006). Heart rate variability: A review. Medical and Biological Engineering and Computing, 44(12), 1031–1051.

    Article  Google Scholar 

  • Andreasson, H., Bouguerra, A., Cirillo, M., Dimitrov, D. N., Driankov, D., Karlsson, L., et al. (2015). Autonomous transport vehicles: Where we are and what is missing. IEEE Robotics Automation Magazine, 22(1), 64–75. https://doi.org/10.1109/MRA.2014.2381357.

    Article  Google Scholar 

  • Antonelli, G., Arrichiello, F., Caccavale, F., & Marino, A. (2014). Decentralized time-varying formation control for multi-robot systems. The International Journal of Robotics Research, 33, 1029–1043.

    Article  Google Scholar 

  • Ball, D., Ross, P., English, A., Patten, T., Upcroft, B., Fitch, R., et al. (2015). Robotics for sustainable broad-acre agriculture. In L. Mejias, P. Corke, & J. Roberts (Eds.), Field and service robotics (pp. 439–453). Cham: Springer.

    Chapter  Google Scholar 

  • Bernardi, L., Wdowczyk-Szulc, J., Valenti, C., Castoldi, S., Passino, C., Spadacini, G., et al. (2000). Effects of controlled breathing, mental activity and mental stress with or without verbalization on heart rate variability. Journal of the American College of Cardiology, 35(6), 1462–1469.

    Article  Google Scholar 

  • Bonarini, A., Mainardi, L., Matteucci, M., Tognetti, S., & Colombo, R. (2008). Stress recognition in a robotic rehabilitation task. In Robotic helpers: User interaction, interfaces and companions in assistive and therapy robotics (pp. 41–48). A workshop at ACM/IEEE HRI.

  • Braezal, C., Dautenhahn, K., & Kanda, T. (2016). Social robotics. In B. Siciliano & O. Khatib (Eds.), Springer handbook of robotics, chap 72 (2nd ed., pp. 1935–1971). New York: Springer.

    Chapter  Google Scholar 

  • Brookhuis, K. A., & de Waard, D. (2010). Monitoring drivers’ mental workload in driving simulators using physiological measures. Accident Analysis & Prevention, 42(3), 898–903.

    Article  Google Scholar 

  • Cacace, J., Caccavale, R., Finzi, A., & Lippiello, V. (2016a). Attentional multimodal interface for multidrone search in the Alps. In IEEE international conference on systems, man, and cybernetics (SMC) (pp. 001178–001183). IEEE.

  • Cacace, J., Finzi, A., & Lippiello, V. (2016b). Implicit robot selection for human multi-robot interaction in search and rescue missions. In Proceedings of IEEE international symposium on robot and human interactive communication (RO-MAN) (pp. 803–808). IEEE.

  • Castaldo, R., Melillo, P., Bracale, U., Caserta, M., Triassi, M., & Pecchia, L. (2015). Acute mental stress assessment via short term HRV analysis in healthy adults: A systematic review with meta-analysis. Biomedical Signal Processing and Control, 18, 370–377.

    Article  Google Scholar 

  • Castaldo, R., Xu, W., Melillo, P., Pecchia, L., Santamaria, L., & James, C. (2016). Detection of mental stress due to oral academic examination via ultra-short-term HRV analysis. In Proceedings of IEEE 38th annual international conference of the engineering in medicine and biology society (EMBC) (pp. 3805–3808). IEEE.

  • Clifford, G. D., Azuaje, F., & McSharry, P. (Eds.). (2006). Advanced methods and tools for ECG data analysis. Norwood: Artech House, Inc,

    Google Scholar 

  • Clifford, G. D., & Tarassenko, L. (2005). Quantifying errors in spectral estimates of HRV due to beat replacement and resampling. IEEE Transactions on Biomedical Engineering, 52(4), 630–638.

    Article  Google Scholar 

  • de la Croix, JP., & Egerstedt, M. (2015). A control lyapunov function approach to human–swarm interactions. In American control conference (ACC), 2015 (pp. 4368–4373). IEEE.

  • Diana, M., de la Croix, JP., & Egerstedt, M. (2013). Deformable-medium affordances for interacting with multi-robot systems. In 2013 IEEE/RSJ International conference on intelligent robots and systems (IROS) (pp. 5252–5257). IEEE.

  • Diaz-Mercado, Y., Lee, S. G., & Egerstedt, M. (2017). Human–swarm interactions via coverage of time-varying densities (pp. 357–385). Cham: Springer.

    MATH  Google Scholar 

  • Dietz, G., Washington, P., Kim, LH., & Follmer, S. et al. (2017). Human perception of swarm robot motion. In Proceedings of the 2017 CHI conference extended abstracts on human factors in computing systems (pp 2520–2527). ACM.

  • Draganjac, I., Miklić, D., Kovačić, Z., Vasiljević, G., & Bogdan, S. (2016). Decentralized control of multi-agv systems in autonomous warehousing applications. IEEE Transactions on Automation Science and Engineering, 13(4), 1433–1447.

    Article  Google Scholar 

  • English, A., Ball, D., Ross, P., Upcroft, B., Wyeth, G., & Corke, P. (2013). Low cost localisation for agricultural robotics. In Proceedings of 2013 Australasian conference on robotics & automation (pp. 1–8). Australian Robotics & Automation Association.

  • Fanti, MP., Mangini, AM., Pedroncelli, G., & Ukovich, W. (2015). Decentralized deadlock-free control for agv systems. In American control conference (ACC), 2015 (pp. 2414–2419). IEEE.

  • Franchi, A., Secchi, C., Son, H. I., Bulthoff, H. H., & Robuffo Giordano, P. (2012). Bilateral teleoperation of groups of mobile robots with time-varying topology. IEEE Transactions on Robotics, 28(5), 1019–1033.

    Article  Google Scholar 

  • Gioioso, G., Franchi, A., Salvietti, G., Scheggi, S., & Prattichizzo, D. (2014). The flying hand: A formation of UAVs for cooperative aerial tele-manipulation. In 2014 IEEE International conference on robotics and automation (ICRA) (pp. 4335–4341). IEEE.

  • Gohara, T., Mizuta, H., Takeuchi, I., Tsuda, O., Yana, K., Yanai, T., et al. (1996). Heart rate variability change induced by the mental stress: The effect of accumulated fatigue. In Proceedings of 15th southern biomedical engineering conference (pp. 367–369). IEEE.

  • Gromov, B., Gambardella, LM., & Di Caro, GA. (2016). Wearable multi-modal interface for human multi-robot interaction. In IEEE international symposium on safety, security, and rescue robotics (SSRR) (pp. 240–245). IEEE.

  • Gunes, H., Nicolaou, M. A., & Pantic, M. (2011). Continuous analysis of affect from voice and face (pp. 255–291). London: Springer.

    Google Scholar 

  • Hocraffer, A., & Nam, C. S. (2017). A meta-analysis of human-system interfaces in unmanned aerial vehicle (UAV) swarm management. Applied Ergonomics, 58, 66–80.

    Article  Google Scholar 

  • Hoover, A., Singh, A., Fishel-Brown, S., & Muth, E. (2012). Real-time detection of workload changes using heart rate variability. Biomedical Signal Processing and Control, 7(4), 333–341.

    Article  Google Scholar 

  • Hornecker, E., & Buur, J. (2006). Getting a grip on tangible interaction: A framework on physical space and social interaction. In Proceedings of the SIGCHI conference on human factors in computing systems (CHI) (pp. 437–446). ACM Press.

  • Kapellmann-Zafra, G., Salomons, N., Kolling, A., & Groß, R. (2016). Human–robot swarm interaction with limited situational awareness. In M. Dorigo, M. Birattari, X. Li, M. López-Ibáñez, K. Ohkura, C. Pinciroli, & T. Stützle (Eds.), Swarm intelligence (pp. 125–136). Cham: Springer International Publishing.

    Chapter  Google Scholar 

  • Kolling, A., Walker, P., Chakraborty, N., Sycara, K., & Lewis, M. (2016). Human interaction with robot swarms: A survey. IEEE Transactions on Human–Machine Systems, 46(1), 9–26.

    Article  Google Scholar 

  • Kulic, D., & Croft, E. A. (2007). Affective state estimation for human–robot interaction. IEEE Transactions on Robotics, 23(5), 991–1000.

    Article  Google Scholar 

  • Lin, CW., & Liu, YC. (2017). Decentralized estimation and control for bilateral teleoperation of mobile robot network with task abstraction. In Proceedings of IEEE international conference on robotics and automation (ICRA) (pp. 5384–5391). IEEE.

  • Luque-Casado, A., Zabala, M., Morales, E., Mateo-March, M., & Sanabria, D. (2013). Cognitive performance and heart rate variability: The influence of fitness level. PloS ONE, 8(2), e56935.

    Article  Google Scholar 

  • Melillo, P., Bracale, M., & Pecchia, L. (2011). Nonlinear heart rate variability features for real-life stress detection. case study: Students under stress due to university examination. Biomedical Engineering Online, 10(1), 96.

    Article  Google Scholar 

  • Mondada, F., Bonani, M., Raemy, X., Pugh, J., Cianci, C., Klaptocz, A., et al. (2009). The e-puck, a robot designed for education in engineering. In Proceedings of 9th conference on autonomous robot systems and competitions (vol. 1, pp. 59–65).

  • Mondada, L., Karim, M. E., & Mondada, F. (2016). Electroencephalography as implicit communication channel for proximal interaction between humans and robot swarms. Swarm Intelligence, 10(4), 247–265. https://doi.org/10.1007/s11721-016-0127-0.

    Article  Google Scholar 

  • Munoz, M. L., van Roon, A., Riese, H., Thio, C., Oostenbroek, E., Westrik, I., et al. (2015). Validity of (ultra-) short recordings for heart rate variability measurements. PloS ONE, 10(9), e0138921.

    Article  Google Scholar 

  • Nagi, J., Giusti, A., Gambardella, LM., & Di Caro, GA. (2014). Human–swarm interaction using spatial gestures. In IEEE/RSJ International conference on intelligent robots and systems (IROS) (pp. 3834–3841). IEEE.

  • Nussinovitch, U., Elishkevitz, K. P., Katz, K., Nussinovitch, M., Segev, S., Volovitz, B., et al. (2011). Reliability of ultra-short ECG indices for heart rate variability. Annals of Noninvasive Electrocardiology, 16(2), 117–122.

    Article  Google Scholar 

  • Podevijn, G., O’Grady, R., Mathews, N., Gilles, A., Fantini-Hauwel, C., & Dorigo, M. (2016). Investigating the effect of increasing robot group sizes on the human psychophysiological state in the context of human–swarm interaction. Swarm Intelligence, 10(3), 193–210. https://doi.org/10.1007/s11721-016-0124-3.

    Article  Google Scholar 

  • Pourmehr, S., Monajjemi, VM., Vaughan, R., & Mori, G. (2013). “you two! take off!”: Creating, modifying and commanding groups of robots using face engagement and indirect speech in voice commands. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 137–142). IEEE.

  • Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., et al. (2009). ROS: an open-source robot operating system. In: Proceeding of ICRA workshop open source software (vol. 3, p. 5).

  • Rani, P., Sarkar, N., Smith, C. A., & Kirby, L. D. (2004). Anxiety detecting robotic system-towards implicit human–robot collaboration. Robotica, 22(1), 85–95.

    Article  Google Scholar 

  • Rani, P., Sims, J., Brackin, R., & Sarkar, N. (2002). Online stress detection using psychophysiological signals for implicit human–robot cooperation. Robotica, 20(06), 673–685.

    Article  Google Scholar 

  • Ren, W., & Beard, R. W. (2005). Consensus seeking in multiagent systems under dynamically changing interaction topologies. IEEE Transactions on Automatic Control, 50(5), 655–661.

    Article  MathSciNet  Google Scholar 

  • Ren, W., & Beard, R. W. (2008). Distributed consensus in multi-vehicle cooperative control: Theory and applications. London: Springer.

    Book  Google Scholar 

  • Rich, C., Ponsler, B., Holroyd, A., & Sidner, CL. (2010). Recognizing engagement in human–robot interaction. In 5th ACM/IEEE international conference on human–robot interaction (HRI) (pp. 375–382). IEEE.

  • Ryu, K., & Myung, R. (2005). Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic. International Journal of Industrial Ergonomics, 35(11), 991–1009.

    Article  Google Scholar 

  • Sabattini, L., Aikio, M., Beinschob, P., Boehning, M., Cardarelli, E., Digani, V., et al. (2018). The pan-robots project: Advanced automated guided vehicle systems for industrial logistics. IEEE Robotics Automation Magazine, 25(1), 55–64.

    Article  Google Scholar 

  • Sabattini, L., Secchi, C., Cocetti, M., Levratti, A., & Fantuzzi, C. (2015). Implementation of coordinated complex dynamic behaviors in multi-robot systems. IEEE Transactions on Robotics, 31(4), 1018–1032.

    Article  Google Scholar 

  • Secchi, C., Sabattini, L., & Fantuzzi, C. (2015). Conducting multi-robot systems: Gestures for the passive teleoperation of multiple slaves. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS) Germany: Hamburg.

  • Shcherbina, A., Mattsson, C. M., Waggott, D., Salisbury, H., Christle, J. W., Hastie, T., et al. (2017). Accuracy in wrist-worn, sensor-based measurements of heart rate and energy expenditure in a diverse cohort. Journal of Personalized Medicine, 7(2), 3.

    Article  Google Scholar 

  • Task Force of The European Society of Cardiology & The North American Society of Pacing and Electrophysiology. (1996). Heart rate variability: Standards of measurement, physiological interpretation, and clinical use. European Heart Journal, 17, 354–381.

    Article  Google Scholar 

  • Thayer, J. F., Åhs, F., Fredrikson, M., Sollers, J. J., & Wager, T. D. (2012). A meta-analysis of heart rate variability and neuroimaging studies: Implications for heart rate variability as a marker of stress and health. Neuroscience & Biobehavioral Reviews, 36(2), 747–756.

    Article  Google Scholar 

  • Villani, V., Capelli, B., & Sabattini, L. (2018a). Use of virtual reality for the evaluation of human–robot interaction systems in complex scenarios. In IEEE (ed) 27th IEEE international symposium on robot and human interactive communication (RO-MAN).

  • Villani, V., Sabattini, L., Battilani, N., & Fantuzzi, C. (2016). Smartwatch–enhanced interaction with an advanced troubleshooting system for industrial machines. IFAC-PapersOnLine, 49(19), 277–282.

    Article  Google Scholar 

  • Villani, V., Sabattini, L., Czerniak, J. N., Mertens, A., & Fantuzzi, C. (2018b). MATE robots simplifying my work: Benefits and socio-ethical implications. IEEE Robotics & Automation Magazine, 25(1), 37–45.

    Article  Google Scholar 

  • Villani, V., Sabattini, L., Riggio, G., Levratti, A., Secchi, C., & Fantuzzi, C. (2017a). Interacting with a mobile robot with a natural infrastructure-less interface. In Proceedings of IFAC 20th world congress international federation automation control IFAC IFAC-PapersOnLine.

  • Villani, V., Sabattini, L., Riggio, G., Secchi, C., Minelli, M., & Fantuzzi, C. (2017b). A natural infrastructure-less human–robot interaction system. IEEE Robotics and Automation Letters, 2(3), 1640–1647.

    Article  Google Scholar 

  • Villani, V., Sabattini, L., Secchi, C., & Fantuzzi, C. (2017c). Natural interaction based on affective robotics for multi-robot systems. In International symposium on multi-robot and multi-agent systems (MRS) (pp. 56–62). IEEE.

  • Villani, V., Sabattini, L., Secchi, C., & Fantuzzi, C. (2018c). A framework for affect-based natural human–robot interaction. In IEEE (ed) 27th IEEE international symposium on robot and human interactive communication (RO-MAN).

  • Vollmer, M. (2015). A robust, simple and reliable measure of heart rate variability using relative RR intervals. In Computing in cardiology conference (CinC), 2015 (pp. 609–612). IEEE.

  • Wallen, M. P., Gomersall, S. R., Keating, S. E., Wisløff, U., & Coombes, J. S. (2016). Accuracy of heart rate watches: Implications for weight management. PLoS ONE, 11(5), e0154420.

    Article  Google Scholar 

  • Wilson, G. F., & Russell, C. A. (2003). Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Human factors, 45(4), 635–644.

    Article  Google Scholar 

  • Wilson, P. A., & Lewandowska-Tomaszczyk, B. (2014). Affective robotics: Modelling and testing cultural prototypes. Cognitive Computation, 6, 814–840.

    Article  Google Scholar 

  • Wurman, P. R., D’Andrea, R., & Mountz, M. (2008). Coordinating hundreds of cooperative, autonomous vehicles in warehouses. AI Magazine, 29(1), 9.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Sciences and Methods for Engineering (DISMI), University of Modena and Reggio Emilia, via G. Amendola, 2, Reggio Emilia, Italy

    Valeria Villani, Beatrice Capelli, Cristian Secchi, Cesare Fantuzzi & Lorenzo Sabattini

Authors
  1. Valeria Villani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Beatrice Capelli
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cristian Secchi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Cesare Fantuzzi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lorenzo Sabattini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Valeria Villani.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This is one of the several papers published in Autonomous Robots comprising the Special Issue on Multi-Robot and Multi-Agent Systems.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 42051 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Villani, V., Capelli, B., Secchi, C. et al. Humans interacting with multi-robot systems: a natural affect-based approach. Auton Robot 44, 601–616 (2020). https://doi.org/10.1007/s10514-019-09889-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-019-09889-6

Keywords

Search

Navigation