Abstract
Intuitive interaction is the cornerstone of accurate and effective performance in remote robotic teleoperation. It requires high-fidelity in control actions as well as perception (vision, haptic, and other sensory feedback) of the remote environment. This paper presents Vicarios, a Virtual Reality (VR) based interface with the aim of facilitating intuitive real-time remote teleoperation, while utilizing the inherent benefits of VR, including immersive visualization, freedom of user viewpoint selection, and fluidity of interaction through natural action interfaces. Vicarios aims to enhance the situational awareness, using the concept of viewpoint-independent mapping between the operator and the remote scene, thereby giving the operator better control in the perception-action loop. The article describes the overall system of Vicarios, with its software, hardware, and communication framework. A comparative user study quantifies the impact of the interface and its features, including immersion and instantaneous user viewpoint changes, termed “teleporting”, on users’ performance. The results show that users’ performance with the VR-based interface was either similar to or better than the baseline condition of traditional stereo video feedback, approving the realistic nature of the Vicarios interface. Furthermore, including the teleporting feature in VR significantly improved participants’ performance and their appreciation for it, which was evident in the post-questionnaire results. Vicarios capitalizes on the intuitiveness and flexibility of VR to improve accuracy in remote teleoperation.
Article PDF
Similar content being viewed by others
References
Pisa/IIT SoftHand Software Package. https://github.com/CentroEPiaggio/pisa-iit-soft-hand. Accessed on 28-Feb-2019. [Online] Available: github.com/CentroEPiaggio/pisa-iit-soft-hand
RobCog-IAI: UE4 based Robot Simulator. https://github.com/robcog-iai/URoboSim. Accessed on 28-Feb-2019. [Online] Available: github.com/robcog-iai/URoboSim
Bejczy, A.K.: Virtual reality in robotics. In: 1996 IEEE Conference on emerging technologies and factory automation. ETFA ’96, vol. 1, pp 7–15 (1996)
Bicchi, A., Gabiccini, M., Santello, M.: Modelling natural and artificial hands with synergies. Philos. Trans. R. Soc. Lond. B Biol. Sci. 366(1581), 3153–3161 (2011)
Bimbo, J., Pacchierotti, C., Aggravi, M., Tsagarakis, N., Prattichizzo, D.: Teleoperation in cluttered environments using wearable haptic feedback. In: 2017 IEEE/RSJ International conference on intelligent robots and systems (IROS), pp 3401–3408 (2017)
Bowman, D.A., McMahan, R.P.: Virtual reality: How much immersion is enough? Computer 40(7), 36–43 (2007)
Buss, S.R.: Introduction to inverse kinematics with Jacobian transpose, pseudoinverse and damped least squares methods. Tech. rep., University of California, San Diego USA (2004)
Calzado, J., Lindsay, A., Chen, C., Samuels, G., Olszewska, J. I.: SAMI: Interactive, multi-sense robot architecture. In: 2018 IEEE 22nd International Conference on Intelligent Engineering Systems (INES), pp. 317–322. https://doi.org/10.1109/INES.2018.8523933(2018)
Catalano, M.G., Grioli, G., Farnioli, E., Serio, A., Piazza, C., Bicchi, A.: Adaptive synergies for the design and control of the Pisa/IIT SoftHand. Int J Robotics Res 33(5), 768–782 (2014)
Chen, J.Y.C., Haas, E.C., Barnes, M.J.: Human performance issues and user interface design for teleoperated robots. IEEE Trans Sys Man Cybern Part C (Applications and Reviews) 37(6), 1231–1245 (2007)
Christie, M., Olivier, P., Normand, J.M.: Camera control in computer graphics. Computer Graphics Forum 27(8), 2197–2218 (2008). https://doi.org/10.1111/j.1467-8659.2008.01181.x
Christou, C. G., Aristidou, P.: Steering versus teleport locomotion for head mounted displays. In: De Paolis, L. T., Bourdot, P., Mongelli, A. (eds.) Augmented Reality, Virtual Reality, and Computer Graphics, pp 431–446. Springer International Publishing, Cham (2017)
Crick, C., Jay, G., Osentoski, S., Pitzer, B., Jenkins, O.C.: Rosbridge: ROS for non-ROS users. In: Inproceedings of the 15th International Symposium on Robotics Research (ISRR) (2011)
DeJong, B.P., Colgate, J.E., Peshkin, M.A.: Improving teleoperation: reducing mental rotations and translations. In: IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA ’04. 2004, vol. 4, pp. 3708–3714 (2004)
Fiorini, S.R., Bermejo-Alonso, J., Gonçalves, P., de Freitas, E.P., Olivares-Alarcos, A., Olszewska, J.I., Prestes, E., Schlenoff, C., Ragavan, S.V., Redfield, S., Spencer, B., Li, H.: A suite of ontologies for robotics and automation [industrial activities]. IEEE Robotics & Automation Magazine 24(1), 8–11 (2017). https://doi.org/10.1109/MRA.2016.2645444
Pignaton de Freitas, E., Olszewska, J.I., Carbonera, J.L., Fiorini, S.R., Khamis, A., Ragavan, S.V., Barreto, M.E., Prestes, E., Habib, M.K., Redfield, S., Chibani, A., Goncalves, P., Bermejo-Alonso, J., Sanz, R., Tosello, E., Olivares-Alarcos, A., Konzen, A.A., Quintas, J., Li, H.: Ontological concepts for information sharing in cloud robotics. Journal of Ambient Intelligence and Humanized Computing. https://doi.org/10.1007/s12652-020-02150-4 (2020)
Galvane, Q.: Automatic cinematography and editing in virtual environments. Ph.D. thesis, Grenoble University (2015)
Han, P.H., Hsieh, C.E., Chen, Y.S., Hsiao, J.C., Lee, K.C., Ko, S.F., Chen, K.W., Chou, C.H., Hung, Y.P.: Aoes: Enhancing teleportation experience in immersive environment with mid-air haptics. In: ACM SIGGRAPH 2017 Emerging Technologies, SIGGRAPH ’17. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3084822.3084823 (2017)
Kaknjo, A., Rao, M., Omerdic, E., Robinson, L., Toal, D., Newe, T.: Real-time video latency measurement between a robot and its remote control station: Causes and mitigation. Wirel. Commun. Mob. Comput. 2018(8638019), 19 (2018)
Kamezaki, M., Yang, J., Iwata, H., Sugano, S.: A basic framework of virtual reality simulator for advancing disaster response work using teleoperated work machines. Journal of Robotics and Mechatronics 26(4), 486–495 (2014). https://doi.org/10.20965/jrm.2014.p0486
Kot, T., Novák, P.: Application of virtual reality in teleoperation of the military mobile robotic system taros. Int J Adv Robotic Sys 15(1), 1729881417751,545 (2018). https://doi.org/10.1177/1729881417751545
Krijn, M., Emmelkamp, P.M., Biemond, R., de Wilde de Ligny, C., Schuemie, M.J., van der Mast, C.A.: Treatment of acrophobia in virtual reality: The role of immersion and presence. Behaviour Research and Therapy 42(2), 229–239 (2004). https://doi.org/10.1016/S0005-7967(03)00139-6. http://www.sciencedirect.com/science/article/pii/S0005796703001396
Krotkov, E., Hackett, D., Jackel, L., Perschbacher, M., Pippine, J., Strauss, J., Pratt, G., Orlowski, C.: The DARPA robotics challenge finals: Results and perspectives, pp. 1–26. Springer International Publishing (2018)
Krupke, D., Starke, S., Einig, L., Zhang, J., Steinicke, F.: Prototyping of immersive HRI scenarios. In: Proc. 20th Intl. Conf. on CLAWAR 2017, pp. 537–544 (2017)
Lipton, J.I., Fay, A.J., Rus, D.: Baxter’s Homunculus: Virtual reality spaces for teleoperation in manufacturing. IEEE Robotics and Automation Letters 3(1), 179–186 (2018)
Loup, G., Loup-Escande, E.: Effects of travel modes on performances and user comfort: A comparison between ArmSwinger and teleporting. International Journal of Human-Computer Interaction 35(14), 1270–1278 (2019). https://doi.org/10.1080/10447318.2018.1519164
Mekuria, R., Blom, K., Cesar, P.: Design, implementation, and evaluation of a point-cloud codec for tele-immersive video. IEEE Trans Circuits Sys Vid Technol 27(4), 828–842 (2017)
Melchiorri, C.: Robot Teleoperation, pp 1–14. Springer London, London (2013)
Miguel, B., Andrew, S., Brian, C., Trey, S., Rodrigo, V.: HTC Vive: Analysis and Accuracy Improvement. In: IEEE/RSJ IROS 2018, pp. 2610–2615 (2018)
Milgram, P., Ballantyne, J.: Real world teleoperation via virtual environment modeling. In: Intl. Conf. on Artificial Reality and Tele-existence (ICAT’97) (1997)
Murashov, V., Hearl, F., Howard, J.: Working safely with robot workers: Recommendations for the new workplace. J. Occup. Environ. Hyg. 13(2), D61–D71 (2016)
Murphy, R.R., Tadokoro, S., Kleiner, A.: Disaster robotics, pp. 1577–1604. Springer International Publishing. (2016)
Naceri, A., Chellali, R., Hoinville, T.: Depth perception within peripersonal space using head-mounted display. Presence: Teleoperators and Virtual Environments 20(3). https://doi.org/10.1162/PRES_a_00048 (2011)
Naceri, A., Mazzanti, D., Bimbo, J., Prattichizzo, D., Caldwell, D. G., Mattos, L. S., Deshpande, N.: Towards a virtual reality interface for remote robotic teleoperation. In: 2019 19th International Conference on Advanced Robotics (ICAR), pp. 284–289. https://doi.org/10.1109/ICAR46387.2019.8981649 (2019)
Naceri, A., Moscatelli, A., Chellali, R.: Depth discrimination of constant angular size stimuli in action space: Role of accommodation and convergence cues Frontiers in Human Neuroscience 9(September). https://doi.org/10.3389/fnhum.2015.00511 (2015)
Nowak, K. L., Biocca, F.: The effect of the agency and anthropomorphism on users’ sense of telepresence, copresence, and social presence in virtual environments. Presence 12(5), 481–494 (2003)
Olszewska, J.I., Barreto, M., Bermejo-Alonso, J., Carbonera, J., Chibani, A., Fiorini, S., Gonçalves, P., Habib, M., Khamis, A., Olivares-Alarcos, A., de Freitas, E.P., Prestes, E., Ragavan, S.V., Redfield, S., Sanz, R., Spencer, B., Li, H.: Ontology for autonomous robotics. In: 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp. 189–194. https://doi.org/10.1109/ROMAN.2017.8172300 (2017)
Ouramdane, N., Otmane, S., Mallem, M.: Interaction 3D en Réalité Virtuelle - Etat de l’art. Technique et Science Informatiques 28(8), 1017–1049 (2009). https://doi.org/10.3166/tsi.28.1017-1049. https://hal.archives-ouvertes.fr/hal-00376458
Park, S., Oh, Y., Hong, D.: Disaster response and recovery from the perspective of robotics. https://doi.org/10.1007/s12541-017-0175-4 (2017)
Peppoloni, L., Brizzi, F., Avizzano, C. A., Ruffaldi, E.: Immersive ROS-integrated Framework for Robot Teleoperation. In: 2015 IEEE Symposium on 3D User Interfaces (3DUI), pp. 177–178 (2015)
Pham, Q.C., Madhavan, R., Righetti, L., Smart, W., Chatila, R.: The impact of robotics and automation on working conditions and employment [Ethical, Legal, and Societal Issues]. IEEE Robotics & Automation Magazine 25(2), 126–128 (2018). https://doi.org/10.1109/MRA.2018.2822058
Quigley, M., Conley, K., Gerkey, B.P., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.Y.: ROS: an open-source Robot Operating System. In: IEEE ICRA Workshop on Open Source Software. IEEE (2009)
Rahnamaei, S., Sirouspour, S.: Automatic viewpoint planning in teleoperation of a mobile robot. J Intell Robotic Sys 76, 443–460 (2014)
Rakita, D., Mutlu, B., Gleicher, M.: An autonomous dynamic camera method for effective remote teleoperation. ACM/IEEE International Conference on Human-Robot Interaction, pp 325–333. https://doi.org/10.1145/3171221.3171279 (2018)
Sanchez-Vives, M.V., Slater, M.: From presence to consciousness through virtual reality. Nat Rev Neurosci 6(4), 332–339 (2005)
Sheridan, T.B.: Musings on telepresence and virtual presence. Presence: Teleoperators and Virtual Environments 1(1), 120–126 (1992). https://doi.org/10.1162/pres.1992.1.1.120
Slater, M., Linakis, V., Usoh, M., Kooper, R., Street, G.: Immersion, presence, and performance in virtual environments: An experiment with tri-dimensional chess. In: ACM Virtual Reality Software and Technology (VRST), pp. 163–172 (1996)
Stotko, P., Krumpen, S., Schwarz, M., Lenz, C., Behnke, S., Klein, R., Weinmann, M.: A VR System for Immersive Teleoperation and Live Exploration with a Mobile Robot (November), pp. 3630–3637. https://doi.org/10.1109/iros40897.2019.8968598 (2020)
Theofanidis, M., Sayed, S.I., Lioulemes, A., Makedon, F.: VARM: Using Virtual Reality to Program Robotic Manipulators. In: Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments - PETRA 2017, pp. 215–221 (2017)
Whitney, D., Rosen, E., Ullman, D., Phillips, E., Tellex, S.: ROS reality: A virtual reality framework using consumer-grade hardware for ROS-enabled robots. In: Proc. IEEE IROS 2018, pp 5018–5025 (2018)
Witmer, B.G., Singer, M.J.: Measuring presence in virtual environments: A presence questionnaire. Presence 7(3), 225–240 (1998)
Acknowledgments
We thank Patricia Yáñez Piqueras and Andrea Santangelo for their technical support in both human and robot experiements.
Funding
Open Access funding provided by Istituto Italiano di Tecnologia. This work was funded by INAIL (Istituto nazionale Assicurazione Infortuni sul Lavoro) and by the Lighthouse Initiative Geriatronics by StMWi Bayern (Project X, grant no.5140951), LongLeif GaPa gGmbH (Project Y, grant no. 5140953).
Author information
Authors and Affiliations
Contributions
List of all authors: Abdeldjallil Naceri, Dario Mazzanti, Joao Bimbo, Yonas T. Tefera, Domenico Prattichizzo, Darwin G. Caldwell, Leonardo S. Mattos and Nikhil Deshpande.
All authors contributed to the study conception and design. Technical work was conducted by Abdeldjallil Naceri and Dario Mazzanti. Abdeldjallil Naceri and Nikhil Deshpande designed the user experimental study. Joao Bimbo contributed to robot arm (UR5) control. Yonas T. Tefera contributed to point cloud data streaming and delay measurements. User study, material preparation, data collection and analysis were performed by Abdeldjallil Naceri. The first draft of the manuscript was written by Abdeldjallil Naceri and Nikhil Deshpande. All authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethical Approval
The testing and experimental procedures were approved by the “Ethics Committee of Liguria Region” (Italy), in accordance with the guidelines of the Declaration of Helsinki for research involving human participants
Consent to Participate
Informed written consent was obtained from all participants involved in the study
Consent to Publish
Individuals appeared in the current submission images/videos gave their consent for publication
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Naceri, A., Mazzanti, D., Bimbo, J. et al. The Vicarios Virtual Reality Interface for Remote Robotic Teleoperation. J Intell Robot Syst 101, 80 (2021). https://doi.org/10.1007/s10846-021-01311-7
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10846-021-01311-7