Skill transfer learning for autonomous robots and human–robot cooperation: A survey☆
Introduction
Autonomous robots and human–robot cooperation are becoming more personalized, interactive, and engaging than ever, providing assistance ranging from daily life to manufacturing, healthcare, and transportation [1]. In these applications, robots are desired to have the capabilities of handling tasks autonomously in different environments and interact with human safely. A key enabler to these applications is to design a system with reasoning and learning ability.
Early robotic manipulation or motion behaviors are usually composed of a series of prescribed motion sequences, which cannot adapt to changing and complex environments [2]. However, for complex tasks, pre-programming a robot is not only inefficiency and tedious, but also impracticable, especially if tasks are constantly updated or changed. In addition, traditional programming approaches cannot achieve autonomous behaviors due to the overlook of human actions and external environments. For example, in a household service scenario, it is hard to pre-program the robot’s tasks taking into account all potential human behaviors and indoor environment configurations. To date, few existing robots can easily perform tie shoelaces, cook, or cut hair.
For human grasping and lifting tasks, neurophysiological research reveals that human relies on the detection of discrete mechanical events that occur when grasping, lifting and replacing an object. Such events represent transitions between phases of the evolving manipulation task (e.g., object contact, lift-off, etc.), and provide critical elements required for the sequential control of the task as well as for corrections and parameterization of the task [3]. Even a simple power grasp manipulation task would engage large parts of the human brain [4], requiring sophisticated control processing. Coordinated and graceful lifting patterns observed in adults are not realized until humans are 8–10 years old [5]. Therefore, humans need nearly a decade of daily practice to master this seemingly simple sensorimotor task.
Consequently, robots are desirable to have the abilities of perception, decision-making and learning in a complex and dynamic environment [6]. It is well known that human manipulation behavior essentially relies on the constant exploration and understanding of the relationship between actions and sensory responses. Human usually preserves the skill knowledge learned in the past and utilizes it to help future learning and problem solving. Inspired by human learning and skill transfer process, developing robot functionality with human level capability of perception, planning and control, has always been an essential goal. Similar to human behaviors, robots typically need to physically interact with environments or humans while performing tasks with rich and informative neurophysiological sensory signals, which are all occurring simultaneously with actions. Moreover, there is a relationship between the sensory responses and actions which could be explored to predict and interpret these behaviors. For autonomous robots and human–robot cooperation, skill transfer learning enables robots to retain or utilize the behaviors observed from human as their skills, improve them by practice, and then apply them into new task environments. The main idea of STL is to develop technical solutions by imitating and exploiting the natural models, systems, and processes.
Motivated by this idea, robots gradually gained the ability to automatically generate motion sequences to perform desired tasks according to the characteristics of the environment and the object, e.g., size and weight [7]. In addition, human neurophysiological signals have been adopted to restore human manipulation functionality by using human muscle activity or cerebral cortex to control the movements of different autonomous devices and perform different human-cooperation tasks [8], [9], [10].
Based on STL, autonomous robots and human–robot cooperation have been a key research area in advanced robotics. The STL methods have been widely applied in perception, learning and control, which integrate knowledge from neurophysiological signals [11], cognitive and executive processing [12]. These new skill acquisition mechanisms significantly facilitate the development of robotic systems with desired properties inspired from neurophysiological and human skill learning processes, such as adaptivity, robustness, versatility, and agility [13]. Because of the advantages listed above, STL has attracted great research attention, and becomes a vital tool enabling robots to deal with environment uncertainties. However, the acquisition of human autonomous skills is quite challenging.
There exist excellent surveys in the literature regarding the skill acquisition process in robot area [14], [15], [16], [17], [18], [19]. However, most of them either focus on the robot learning (e.g, machine learning in robotics, RL in robot control) or pure affordances in psychology and neuroscience. Few of them reviews the topic of STL, especially for the skill acquisition via neurophysiological signals. In this paper, we summarize the status and challenges of STL systems. The remainder of this paper is organized as follows. In Section 2, we present the state of the art of STL which includes the categorizations, framework and application of STL. Section 3 introduces the robot learning of STL. Section 4 reviews the recent developments of STL via neurophysiological signals. Section 5 discusses future directions for STL in autonomous robots and human–robot cooperation.
Section snippets
Skill transfer learning
Autonomous robots and human–robot cooperation are desirable to handle objects of different size and weight in a dynamic environment. Skill learning cannot only address challenges caused by the lack of accurate object model and interaction dynamic model, but also the increasing complexity of perception and control of systems with large degrees of freedom. In addition, with the development of recent neuroscience technologies, precise nature of human neural representations can be utilized to
Skill transfer based on robot learning
Robot learning can implicitly train a robot, so that human users can minimize or eliminate explicit tedious programming of tasks. Most of robot learning methods are data-driven. The data required for robot learning can be generated by interactions between the robot and the environment or provided by experts. Based on this idea, the robot learning for skill acquisition can be classified in the following types.
Skill transfer learning via neurophysiological signals
Since human neurophysiology contains rich and useful environmental information, they can facilitate robot skill transfer. Meaningful tactile feedbacks or perceptions have been studied in a large number of neurophysiological and behavioral researches. Human is able to understand the internal mechanism of actions and then utilize it to realize appropriate behavior [12]. Inspired by these sensorimotor abilities, detecting environment through a neuromorphic interface and initiating an automated
Discussion and open questions
As for human skill acquisition, to be able to acquire manipulation skills from human, robots should have the ability to learn behaviors from autonomous perception and control. STL can address uncertain models of manipulated objects and robot dynamics, as well as complex robotic systems with a large number of DOFs. Moreover, STL could utilize a user’s motivation and cognitive arousal to achieve skill transfer directly. Many of the proposed STL approaches to skill acquisition for robot either
Conclusion
In conclusion, an overview on the current state of the art of STL research has been presented. In addition to discussing STL, various other robot learning algorithms are discussed and compared. We also include neurophysiological skill acquisition using STL. In the end, we present the challenges and open questions of applying STL to autonomous robot and human-cooperation.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Yueyue Liu received the M. S. degree in control engineering from the South China University of Technology, Guangzhou, China, in 2017. Now he is currently pursuing the Ph.D. degree with the College of Automation Science and Engineering. His research interests include mobile manipulation and autonomous robot.
References (123)
Is imitation learning the route to humanoid robots
Trends Cogn. Sci.
(1999)- et al.
Iterative learning of grasp adaptation through human corrections
Robot. Auton. Syst.
(2012) - et al.
Learning human arm movements by imitation:: Evaluation of a biologically inspired connectionist architecture
Robot. Auton. Syst.
(2001) - et al.
Cooperative manipulation for a mobile dual-arm robot using sequences of dynamic movement primitives
IEEE Trans. Cogn. Dev. Syst.
(2018) - et al.
Combining active learning and reactive control for robot grasping
Robot. Auton. Syst.
(2010) - et al.
Autonomous grasp and manipulation planning using a ToF camera
Robot. Auton. Syst.
(2012) - et al.
Interaction learning for dynamic movement primitives used in cooperative robotic tasks
Robot. Auton. Syst.
(2013) - et al.
Learning CPG-based biped locomotion with a policy gradient method
Robot. Auton. Syst.
(2006) - et al.
Human–robot skills transfer interfaces for a flexible surgical robot
Comput. Methods Programs Biomed.
(2014) - et al.
Human-inspired control of dual-arm exoskeleton robots with force and impedance adaptation
IEEE Trans. Syst. Man Cybern.
(2018)
High-performance neuroprosthetic control by an individual with tetraplegia
Lancet
2008 special issue: Reinforcement learning of motor skills with policy gradients
Neural Netw.
The new robotics—towards human centered machines
HFSP J.
Off-line programming of an industrial robot for manufacturing
Int. J. Adv. Manuf. Technol.
Tactile sensory control of object manipulation in human
An approach to automatic robot programming based on inductive learning
Int. J. Robot. Res.
Brain-computer interface-based stochastic navigation and control of a semiautonomous mobile robot in indoor environments
IEEE Trans. Cogn. Dev. Syst.
Adaptive neural control of a kinematically redundant exoskeleton robot using brain-machine interfaces
IEEE Trans. Neural Netw.
Motor cortical activity changes during neuroprosthetic-controlled object interaction
Sci. Rep.
Shared human–robot proportional control of a dexterous myoelectric prosthesis
Nat. Mach. Intell.
Neurophysiological mechanisms underlying the understanding and imitation of action
Nat. Rev. Neurosci.
Self-organization, embodiment, and biologically inspired robotics
Science
Reinforcement learning in robotics: A survey
Int. J. Robot. Res.
Optimal and autonomous control using reinforcement learning: A survey
IEEE Trans. Neural Netw.
A survey on policy search algorithms for learning robot controllers in a handful of trials
Combined sensing, cognition, learning, and control for developing future neuro-robotics systems: A survey
IEEE Trans. Cogn. Dev. Syst.
Affordances in psychology, neuroscience, and robotics: A survey
IEEE Trans. Cogn. Dev. Syst.
Perspectives on deep multimodel robot learning
A learning-based hierarchical control scheme for an exoskeleton robot in human-robot cooperative manipulation
IEEE Trans. Cybern.
On learning, representing, and generalizing a task in a humanoid robot
IEEE Trans. Syst. Man Cybern. B
Phase estimation for fast action recognition and trajectory generation in human–robot collaboration
Int. J. Robot. Res.
Keyframe-based learning from demonstration method and evaluation
Int. J. Soc. Robot.
Teleoperation control based on combination of wave variable and neural networks
IEEE Trans. Syst. Man Cybern. A
Teaching physical collaborative tasks: Object-lifting case study with a humanoid
An enhanced teaching interface for a robot using DMP and GMR
Int. J. Intell. Robot. Appl.
Development of multi-fingered dexterous hand for grasping manipulation
Sci. China F
Development of sensory-motor fusion-based manipulation and grasping control for a robotic hand-eye system
IEEE Trans. Syst. Man Cybern. A
SEMG-based joint force control for an upper-limb power-assist exoskeleton robot
IEEE J. Biomed. Health Inf.
Learning collaborative manipulation tasks by demonstration using a haptic interface
An experience-driven robotic assistant acquiring human knowledge to improve haptic cooperation
Learning contact-rich manipulation skills with guided policy search
Unifying perception, estimation and action for mobile manipulation via belief space planning
Object search by manipulation
Efficient planning in non-Gaussian belief spaces and its application to robot grasping
Reinforcement learning of manipulation and grasping using dynamical movement primitives for a humanoidlike mobile manipulator
IEEE/ASME Trans. Mechatronics
Cited by (0)
Yueyue Liu received the M. S. degree in control engineering from the South China University of Technology, Guangzhou, China, in 2017. Now he is currently pursuing the Ph.D. degree with the College of Automation Science and Engineering. His research interests include mobile manipulation and autonomous robot.
Zhijun Li received the Ph.D. degree in mechatronics, Shanghai Jiao Tong University, P. R. China, in 2002. From 2003 to 2005, he was a postdoctoral fellow in Department of Mechanical Engineering and Intelligent systems, The University of Electro-Communications, Tokyo, Japan. From 2005 to 2006, he was a research fellow in the Department of Electrical and Computer Engineering, National University of Singapore, and Nanyang Technological University, Singapore. From 2017, he is a Professor in Department of Automation, University of Science and Technology, Hefei, China. From 2019, he is the Vice Dean of School of Information Science and Technology, University of Science and Technology of China, China.
From 2016, he has been the Co-Chairs of IEEE SMC Technical Committee on Bio-mechatronics and Bio-robotics Systems (B2S), and IEEE RAS Technical Committee on Neuro-Robotics Systems. He is serving as an Editor at-large of Journal of Intelligent & Robotic Systems, and Associate Editors of several IEEE Transactions. Dr. Li’s current research interests include wearable robotics, tele-operation systems, nonlinear control, neural network optimization, etc.
Huaping Liu received the Ph.D. degree from Tsinghua University, Beijing, China, in 2004. He is currently an Associate Professor with the Department of Computer Science and Technology, Tsinghua University. His research interests include robot perception and learning. Dr. Liu served as a Program Committee Member for RSS2016 and IJCAI2016. He serves as an Associate Editor for several journals including IEEE ROBOTICS AND AUTOMATION LETTERS, Neurocomputing, Cognitive Computation, and some conferences including the International Conference on Robotics and Automation and the International Conference on Intelligent Robots and Systems.
Zhen Kan received the Ph.D. degree in mechanical and aerospace engineering from the University of Florida, Gainesville, FL, USA, in 2011. He is a Professor with the Department of Automation, University of Science and Technology of China, Hefei, China. He was a Postdoctoral Research Fellow with the Air Force Research Laboratory, Eglin AFB, FL, USA, and the University of Florida REEF, Shalimar, FL, USA, from 2012 to 2016, and an Assistant Professor with the Department of Mechanical Engineering, University of Iowa, Iowa City, IA, USA. His current research interests include networked robotic systems, Lyapunov-based nonlinear control, graph theory, complex networks, and human-assisted estimation, planning, and decision making. Prof. Kan currently serves as an Associate Editor on the Conference Editorial Board of the IEEE Control Systems Society and Technical Committee for several internationally recognized scientific and engineering conferences.
- ☆
This work was supported in part by the National Natural Science Foundation of China under Grant 61625303, and Grant 61751310, in part by the National Key Research and Development Program of China under Grant 2017YFB1302302, Grant 2018YFC2001600, in part by the Anhui Science and Technology Major Program under Grant17030901029.