Development of Human Support Robot as the research platform of a domestic mobile manipulator

Abstract

There has been an increasing interest in mobile manipulators that are capable of performing physical work in living spaces worldwide, corresponding to an aging population with declining birth rates with the expectation of improving quality of life (QoL). We assume that overall research and development will accelerate by using a common robot platform among a lot of researchers since that enables them to share their research results. Therefore we have developed a compact and safe research platform, Human Support Robot (HSR), which can be operated in an actual home environment and we have provided it to various research institutes to establish the developers community. Currently, the number of HSR users is expanding to 44 sites in 12 countries worldwide (as of November 30th, 2018). To activate the community, we assume that the robot competition will be effective. As a result of international public offering, HSR has been adopted as a standard platform for international robot competitions such as RoboCup@Home and World Robot Summit (WRS). HSR is provided to participants of those competitions. In this paper, we describe HSR’s development background since 2006, and technical detail of hardware design and software architecture. Specifically, we describe its omnidirectional mobile base using the dual-wheel caster-drive mechanism, which is the basis of HSR’s operational movement and a novel whole body motion control system. Finally, we describe the verification of autonomous task capability and the results of utilization in RoboCup@Home in order to demonstrate the effect of introducing the platform.

Introduction

Aging society with the declining birthrate has become a serious social problem in Japan, and many other countries have the same problem. As society’s aging rapidly advances, the shortage of workers and care workers has become a major issue [1]. To improve quality of life (QoL), it is very important to promote the independence of people with disabilities and the elderly as well as provide household support to these groups and the general population. It is considered that developing a robot which works on behalf of a human is one of possible solutions.

The Human Support Robot (HSR) discussed in this paper is a domestic mobile manipulator robot which holds both functions of physical work and communication [2, 3], and we aim to establish HSR Developers Community as a strong network between various research institutes with sharing the same robot platform in order to accelerate the research and development of a domestic mobile manipulator (Fig. 1).

Fig. 1

HSR developers community

Figure 2 shows the appearance of HSR. HSR has been in development with the goal of conducting tasks such as operating on furniture (i.e. opening/closing drawers, using microwaves, etc...), fetch and carry of daily necessities, and tidying rooms up. It aims to support people having greater needs for daily life (Fig. 3).

Fig. 2

Human Support Robot (HSR) [2]. a Normal posture, b Extended posture

Fig. 3

Utilization of HSR [19]. a Independent living support, b Remote care support, c Housework support

In order to achieve the realization of a robot which performs physical work at home, it requires tremendous development of software for executing tasks in a real environment in addition to the hardware which may coexist with people in their living space. In order to make it possible to operate in a real environment, HSR was developed as a research platform with a compact body that has both capabilities and safety for field tests in home environments.

In recent years, international robot competitions have attracted attention as an effective approach to accelerate research and development of robots [4,5,6]. HSR was adopted as the standard platform for RoboCup@Home from the six candidates who passed the document selection under the review of the RoboCup International Committee in 2016 [7]. HSR has been used at the Domestic Standard Platform League (DSPL) for home service robots since RoboCup 2017 Nagoya. Moreover, it has been adopted as a standard platform for the service robot competition of the World Robot Summit (WRS) [6] which is scheduled to be held in 2020 in Japan after the Tokyo Olympic Games. Given these facts, it seems that HSR has gained popularity in robot competitions. HSR was provided to 44 universities and companies in 12 countries through public offerings as of November 30th 2018, and research and development is proceeding on each projects.

In this paper, we provide information on development history, aims and development technologies as a reference for researchers interested in HSR. The remainder of this paper is structured as follows. In "Related work" section, we discusses related work. In "Concept and design" section, we mainly explain the hardware of HSR, including development history. It broadly shows that the hardware of HSR is able to operate on quite a broad range of tasks despite the simplicity of its design. One of the goals of HSR is to realize autonomous operation of those tasks. In "Software architecture" section, we show the architecture of software and describe the newly developed whole-body control and operational plan to utilize of the structure of HSR in detail. In "Results and discussion" section, we show the verification of the autonomous task capability and the basic functions as the research platform. In addition we briefly describe the utilization of the HSR in RoboCup 2017 in Nagoya and RoboCup 2018 in Montreal, and we examine the effect of HSR adoption as a platform. Finally, we present the conclusions of this paper in "Conclusion" section.

Related work

In recent years, there has been increasing interest in mobile manipulators that are capable of performing physical work in home environments. Several research groups have developed their own mobile manipulators, for example [8,9,10,11]. Also, there have been several attempts to build commercially available robot platforms designed for mobile manipulation research [12,13,14,15,16]. They have been distributed to several research institutes worldwide and utilized for a variety of research. However, lower weight and dimensions are required in actual home environments, as mentioned in “Research platform HSR development” section. In addition, it is necessary to reduce weight to lower the potential risk of collision accidents. Considering such requirements, HSR is carefully developed to be safe and compact in order to fit into realistic home environments while keeping sufficient physical capability to perform useful tasks.

Concept and design

Needs survey by prototype

Our first prototype named Delivery Robot, which was exhibited in the conference of the Japan Circulation Society in 2006, was the concept model developed for hospital support (Fig. 4).

Fig. 4

Delivery Robot in 2006 [19]

This robot had the similar features of HSR with a single arm and wheels. When we started designing the next prototype, we reconsidered what was necessary to implement robots in actual home environments. At first we referred service dogs, which were trained to support people with physical disabilities, with strong needs of living support.

Service dogs are able to understand simple orders such as “Take PET bottle” or “Open door” and execute the tasks. According to Japan Service Dog Association (JSDA) [17], main tasks of service dogs are defined as follows:

• 1. Picking up dropped objects (i.e. coins, cards, keys, documents, etc.).

• 2. Retrieving objects (i.e. pet bottles, wheel chairs, parking tickets, etc.).

• 3. Enabling contact during emergencies (i.e. phone calls, family, emergency buttons, etc.).

As of the beginning of July 2012, the number of service dogs was only 59 (66 as of October, 2018) in Japan, which was not sufficient for latent demand [18]. One of the reasons is that the owners must be capable of caring for service dogs, such as feeding and treating feces. They are not necessarily easy work for people with disabilities. If the care for a dog is physically difficult or not preferred, there is the possibility that a small mobile manipulator with autonomous and remote control function could be substituted.

To investigate actual demands, we conducted questionnaires on potential service dog users in collaboration with JSDA. As a result, we made sure that demands for fetch and carry tasks and remote assistance were high (Fig. 5) as same as the main tasks above.

Fig. 5

Results of questionnaire investigated with JSDA

As our first trial, our goal was to realize tasks with robots only for indoors, considering technical difficulty. As for emergency support, we expected that we could deal with it by attaching devices such as camera, sensor and wireless. In response to picking and retrieving in a house , we decided to develop a compact and safe mobile manipulator.

To study it, we made a prototype with a 7 degrees-of-freedom (DoF) arm and a mobile base, based on differential drive in 2012 and gathered opinions from target users and experts as described in [19] (Fig. 6). As stated in the paper above, we got 33 comments from two subjects with disabilities in the limbs. These comments could be mainly divided into opinions related to robot software or HMI and opinions related to mechanical design. Taking into account the next hardware design, we applied the opinions connected with mechanical design. The related comments described in [19] are extracted and described below.

• 1. HSR cannot reach objects that the subject herself cannot reach (during autonomous Fetch task).

• 2. Not knowing how far the HSR will extend its arm to hand over items is scary.

• 3. Time necessary to complete tasks is too long (HSR is too slow).

• 4. Even with only limited autonomy, HSR feels like a companion. Happy to have HSR around the house.

• 5. Increase maximum payload (lift heavier items like a dictionary).

• 6. Retrieve objects which the subject cannot reach (e.g., on the high shelf around 1700[mm], in the back of the refrigerator).

• 7. Be able to lift heavy objects (20[kg]) even if only momentarily

• 8. Clean a room (vacuum functionality).

• 9. Assist subject in getting in and out of a wheelchair.

• 10. Keep the robot size small, even if it means sacrificing functionality.

• 11. Possible to be used outdoors, climb slopes and stairs.

Fig. 6

Examples of prototype tests in [19]. a Grasping bottle, b Support of drinking

Regarding the opinions 1, 5, 6, 7, 8, 9 and 11, as a result of considering the opinions 4 and 10, we decided prioritize the compact and safe physique without making major changes in function.

Regarding the opinion 3, as a result of analyzing the task, the stopping time of the arm and the wheels was 61%. Therefore we thought that the planning software of the movement should be improved. At the same time, we considered that it could be one of the fundamental problems of the mechanical design. Therefore, we discussed 3, and additionally 2, with the experts of Yokohama Rehabilitation Services and we concluded that the robot’s motion appears to be complicated and unexpected, because the arm starts moving after the wheels completely stop, resulting that it gives users uneasy feeling and impression of slow motion. Taking it into account, we started new simpler design based on coordinated movements of the wheels and the arm.

Preliminary mock-up examination

For the new simpler design, we tried to change from the 7 DoF arm with a differential drive type mobile base to a less capable 4 DoF arm with an omnidirectional mobile base. We first made a mock-up and evaluated the function of the hardware by manipulating it remotely (Fig. 7).

Fig. 7

Object handling by mock-up [46]. a Floor, b Kitchen

It was investigated whether or not it could tidy up a very cluttered environment by teleoperation (left side in Fig. 8). We discovered the robot was able to accomplish the task (right side in Fig. 8). In addition, we confirmed that the robot could clean up the floor, open and close the drawer, and approach the kitchen sink.

Fig. 8

Tidy up by mock-up [46]. a Before, b After

Although this robot consists of a simple mechanism, it is possible to realize practical fetch-and-carry and cleanup tasks at home if the capabilities of recognition and intelligent decision-making are improved.

Research platform HSR development

Based on the previous studies, we describe the design of the proposed platform in the following.

Regarding the targets of the design, we set the maximum payload weight in arbitrary posture to 1.2 kg in order to grasp 43 classes of [20] from floor to desk (0–725 mm) in three directions (top, side, front) of the hand. The target height is set to 1.35 m considering the reachability to furniture of shoulder height (1331 mm, referred from [21]), which could be normally accessed by general people standing. The maximum velocity is set to 0.8 km/h based on sensory evaluations using the previous prototype, considering real field tests to elderly people or people with disabilities, while giving them a sense of security. Based on ISO-7176/5 or JIS-T 9203, the width of the electric wheelchair is set to 700 mm or less. Also, from the architectural design standards determined by the Ministry of Land, Infrastructure, Transport and Tourism (MLIT) [22], the width required for a wheelchair and a person facing the sideways to pass each other is set to 120 cm or more. Based on these conditions, the target width is set to less than 50 cm with a cylindrical telescoping body, taking into account the parallel running and passing with the wheelchair. In addition, we set targets with a step difference of 5 mm and a slope of 5° from the barrier free standard in Japan [23]. Moreover the height of the robot’s standard posture is set to less than the eye level of the wheelchair user (110 cm [24]), because we do not want to give the sense of intimidation to wheelchair users. Considering experiments under development in actual fields with people, we set the maximum kinetic energy 10 J or less with reference to ISO 14120.

Next, we describe actual results of the design. Figure 9 shows the joint configuration of HSR. Here, the shoulder extends twice the length of the head by the movement of #6. Regarding motors, 40, 20, 14, 14, 14, 14, 25, 14, 14, 25, and 25 W (rated output) of maxon motor© for joints #0 to #10.

Fig. 9

Joint Configuration of HSR

In order to achieve light weight, aluminum alloy is mainly adopted as the structural material, whereas iron is adopted as a part connecting the arm and the body that are subjected to the most load. For the # 1 axis with a heavy weight burden, springs are arranged in parallel to compensate for a load of 40 % or more. When the object of 1.2 kg is grasped, the maximum tip displacement of the whole arm is 3.4 mm, and it is judged that there is no big influence on object grasping. As shown in Figs. 10 and 11, it is designed to meet the requirements of the size. In addition, it is designed to be able to handle from a floor to a desk by gripping postures in three directions as shown in Figs. 12 and 13. The basic specifications of HSR are shown in Table 1.

Fig. 10

Side view of HSR, person standing and riding wheelchair

Fig. 11

Top View of wheelchair and HSR

Fig. 12

Picking from floor in three directions

Fig. 13

Picking From Desk in Three Directions

Table 1 HSR basic specifications [3]

Figure 14 shows the sensors installed in the HSR. Various sensors are installed for the ease of use as a research platform.

Safety measures include reduction of pinch points by simplification of the arm, reduction of fall hazard through gravity compensation, reduction of contact danger by driving thrust reduction and magnetic tape stop function to reduce the risk of fall from stairs and steps. It implements force control with a 6-axis force-torque sensor on the wrist, and compliance control using joint modules based on series elastic actuators [25] that utilize the elasticity of the timing belts for #0 and #1. Regarding computational resources, CPU board (Intel©Core™i7-4700EQ CPU 2.4GHz) and GPU board (NVIDIA© Jetson™) are mounted inside HSR. When a large-scale calculation is required, it is possible to use an external server via wireless or wired LAN.

Fig. 14

Sensors and equipments of HSR [3]

As other features, it has installed a suction pad with suction force 3 N to suck thin objects like mail, bill, credit card and coin which are difficult to grasp by hand.

The robot as a whole has 8 DoF for manipulation, comprised of the 3 DoF of the mobile base, 4 DoF of the arm, and 1 DoF of the torso lift. Thus, it is possible to generate flexible movement by moving the mobile base and the arm together. We developed a novel whole-body motion control method making better use of the configuration of this robot for coordination between transportation movement and the grasping operation as described later.

To clarify the features of HSR, we compare it with the existing platforms in [12] and [15] used by multiple research institutes. Here, we call the robots described in papers [12] and [15] as RA and RB respectively. First, the payload weights of RA, RB and HSR are 1.8 kg, 3 kg and 1.2 kg, respectively. Note that RA has 2 arms and it could hold 3.6 kg objects. Although there are differences in proportion to the physique, every robot has a payload that can grasp all our target objects in 43 classes of [20]. Regarding the size, the widths of RA and RB are 66.8 cm and 54 cm, respectively, which do not fit into the above-mentioned width of 50 cm. On the other hand, the HSR is 43 cm in width, which is smaller than the target size meaning that it runs parallel to a wheelchair. Regarding the kinetic energy, as a result of calculation with the maximum speed and weight being disclosed, RA is 113 J and RB is 35 J, both of which exceed the target value 10 J. With regard to HSR, considering the worst case, we calculate the kinetic energy with the maximum speed of 0.36 m/s from the no-load rotation speed of the wheel motor, and the weight of 50 kg assuming to add 13 kg of additional devices. It is 3.24 J, which is lower than the target value 10 J. These results has clarified the features of HSR.

Software architecture

Strategy for development environment of software architecture

HSR’s software architecture is built on ROS (Robot Operating System) [26]. The overview of the system architecture is shown in Fig. 15.

Fig. 15

Software architecture of HSR [47]

The software system is mainly divided into four subsystems: (1) the device control subsystem, which is structured by a group of servo amplifiers, (2) the motion control subsystem, which operates in real time on the robot computer, (3) the higher-level functional subsystem, which consists of ROS node groups also operates on the robot computer, and (4), the user interface subsystem.

Device control subsystem

A group of servo amplifiers receive commands from the motion control subsystem through an exclusive, master-slave type communication protocol via RS485 and perform motor control.

Motion control subsystem

The motion control subsystem performs the real-time control in the user space with a Linux kernel for HSR to which the PREEMPT_RT patch has been applied. In the core process of communication with various devices and cycle control, a plug-in mechanism provided by ros_control [27] is implemented and handles the dynamic loading of the cycle control plug-in. In addition to the standard plug-in support of ros_control, we developed controllers for unique hardware such as the omnidirectional base mechanism and the gripper, as well as a plug-in which provides a low-level interface to manipulate devices. The diagnostic information of each device (diagnostics), such as sensors and actuators, is integrated into ROS diagnostic aggregator.

Functional subsystem

The higher-level functional subsystem has access to the robot hardware via the ROS interface provided by those plug-ins. In the higher-level functional subsystem, we developed our own ROS packages to perform the operations of recognition, autonomous movement, manipulation, and teleoperation.

User interface subsystem

In addition, the following mechanisms are introduced in those ROS interfaces and package groups to make it more user-friendly.

Whilst adoption of ROS has greatly contributed to robot software development, there has been concern that it has increased the learning cost for entry-level users, with advanced concepts such as the publish-subscribe communication model or callback programming. This is based on our experience and observations gained through actual in-house training and collaborative research. To lower the learning cost, HSR provides a high-level programming interface written in Python [28]. It abstracts the ROS interface of the functional subsystem and enables highly abstracted robot programming in the form of imperative operations on objects. Listing 1 shows an example program which causes HSR to grasp a bottle at a given position. In addition, an interactive shell based on IPython [29] allows users to access advanced editing tools such as tab completion or runtime query against live objects. Also, it supports exploratory programming, so a sequence of commands can be incorporated into the task program. Through this, it is expected that robot program prototyping speed will increase.

Whole-body cooperative manipulation system

There is a wealth of research about omnidirectional mechanisms [30]. For HSR, we chose a dual-wheel caster-drive mechanism [31]. A conceptual diagram of this mechanism is shown in Fig. 16.

Fig. 16

Dual-wheel caster-drive mechanism [3]

Jacobian J is denoted by (1) assuming that the radius of the wheel is r, the angle of the pivot axis connecting the top table is ${\theta _H}$, the tread of the wheels is W, and the distance from the center axis to the axle center is H.

$$\begin{aligned} J = \left[ \begin{array}{ccc} \frac{r}{2} \cos {\theta _H}-\frac{rH}{W} \sin {\theta _H} &{} \frac{r}{2} \cos {\theta _H}+\frac{rH}{W} \sin {\theta _H} &{} 0 \\ \frac{r}{2} \sin {\theta _H}+\frac{rH}{W} \cos {\theta _H} &{} \frac{r}{2} \sin {\theta _H}-\frac{rH}{W} \cos {\theta _H} &{} 0 \\ \frac{r}{W} &{} -\frac{r}{W} &{} 1 \end{array} \right] \end{aligned}$$

(1)

Using J, the velocity of the right and left wheels and the rotation axis ${\omega _R, \omega _L, \omega _H}$ is determined from the velocity of the body ${\dot{x},\dot{y},\theta }$ as represented in the following.

$$\begin{aligned} \left[ \begin{array}{ccc} \omega _R \\ \omega _L\\ \omega _H \end{array} \right] = J^{-1}\left[ \begin{array}{ccc} \dot{x} \\ \dot{y}\\ \dot{\theta } \end{array} \right] \end{aligned}$$

(2)

Since J is full rank regardless of the state, this mechanism is a holonomic mechanism which can always generate speed in all directions.

As mentioned in [31], this method precludes problems such as vibration, low-load capacity, and the over-constraint, in contrast to other omnidirectional mechanisms using special wheels such as [32, 33]. Less vibration and less entanglement of electric cables are excellent characteristics for a mobile manipulator for home use. HSR is able to position its end effector by combining its omnidirectional base, 4 DoF from its arm, and 1 DoF from its torso lift.

The following describes the manipulation dataflow (Fig. 17).

Fig. 17

Manipulation dataflow diagram [3]

Object recognition

The position and orientation of the recognized objects are obtained from the camera, depth sensor, etc. Depending on end use, this is done by using a marker and/or local-feature-based recognition.

Grasp pose generation

Let a sequence of n poses of the end effector be $\mathbf{T_e} = \{ T_{e1}, T_{e2}, \dots T_{en} \}$, where $T_{ei}$ is the candidate at the order of i of the position and orientation in which an object can be grasped after moving the end-effector. As can be seen, multiple position, orientation candidates exist, from which even a single object can be grasped.

Motion planning

This computes the geometric whole-body trajectory ${\varvec{\theta }}$, which includes the base movement and collision avoidance with external objects and itself as it navigates to any one of the candidate position and orientation sequences, $\mathbf{T_e}$. In sampling-based motion planning algorithms, it is necessary to solve inverse kinematics repeatedly. We developed a fast hybrid inverse kinematics algorithm that combines analytical and numerical approaches to accelerate motion planning described in "Hybrid inverse kinematics” section.

Motion execution

The geometrical whole-body trajectory is converted into the timestamped trajectory by time-optimal path parameterization. In this case, the travel distance v is acquired using laser odometry in order to ensure accuracy, and it provides feedback of its position.

Extending CBiRRT2

In motion planning, constraints and termination conditions in the task space are given to the redundant manipulator, and a high-speed solution method, CBiRRT2 [34] is used. CBiRRT2 is a variant of bi-directional RRT (Rapidly-exploring Random Tree) algorithm capable of efficiently searching a configuration space and the following expansions using a spatial expression called a TSR (Task Space Region).

• 1. By expressing the termination condition using TSR, it populates additional goal configurations during the search.

• 2. By expressing the constraint condition using TSR, it constrains samples during the search.

The following describes the extension of CBiRRT2. In an omnidirectional mobile manipulation system, there are two different types of demands for motion planning. One is to move the hand accurately to a goal, and the other is to exert a large force on objects. In order to reach a goal accurately, it is effective to move the hand as much as possible while limiting the motion of the mobile base. On the other hand, to exert a large force, it is effective to prioritize the motion of the mobile base.

Arm priority operation and base priority operation are used for inverse kinematics and motion planning, and the distance space of the configuration space is weighted as necessary.

Figure 18 shows the result of planning the trajectory that moves the end effector along a straight line (a) or the trajectory to move along the circular arc (b) by coordinating the degrees of freedom of the base of the HSR using the motion planning unit.

Fig. 18

Whole body planning with priority [3]. a Arm priority, b Base priority

Hybrid inverse kinematics

This part explains the inverse kinematics that are required to be calculated many times during motion planning in CBiRRT2. For inverse kinematics during motion planning, it is preferable to use an analytical solution rather than a numerical solution. A numerical solution tends to have high initial value dependence, high computational cost, and a local minimum issue. HSR obtains an analytical solution for 8 DoF (5 arm DoF + 3 base DoF), and a globally optimized solution is derived by further numerical calculation.

First, we found an inverse kinematics analytic solution f in (3) that realizes the tip pose ${\varvec{T}}_e$. This configuration is shown in Fig. 19.

Fig. 19

HSR’s joint configuration for IK [3]

$\theta _0, \theta _1, \theta _2$ are the degrees of freedom of the base, represented by Cartesian coordinates. $\theta _3, \theta _4, \theta _5, \theta _6, \theta _7$ are the degrees of freedom of the arm. ${\varvec{T}}_e$ is the reference value of the tip pose. Since the degrees of freedom are redundant, we calculate $\theta _2, \theta _4$ as parameters. Using these two joints as parameters makes the calculation easiest.

$$\begin{aligned} (\theta _0, \theta _1, \theta _3, \theta _5, \theta _6, \theta _7) = f({\varvec{T}}_e, \theta _2, \theta _4) \end{aligned}$$

(3)

Next, the optimization procedure is shown. In this case, the solution joint angles ${\varvec{\theta }} = (\theta _0, \theta _1, \theta _2, \theta _3, \theta _4, \theta _5,$ $\theta _6, \theta _7)$ are derived as closest to reference joint angles ${\varvec{\theta }}^{ref} = (\theta _0^{ref}, \theta _1^{ref}, \theta _2^{ref}, \theta _3^{ref}, \theta _4^{ref}, \theta _5^{ref}, \theta _6^{ref}, \theta _7^{ref})$, where ${\varvec{\theta }}^{ref}$ is arbitrarily set. The evaluation function is defined as follows:

$$\begin{aligned} V(\theta _2, \theta _4, {\varvec{T}}_e, {\varvec{W}}, {\varvec{\theta }}^{ref}) = || {\varvec{W}}({\varvec{\theta }}^{ref} - {\varvec{\theta }}) || \end{aligned}$$

(4)

where ${\varvec{\theta }}$ is the total joint angle by analytical solution of the joint angles obtained by (3) and $\theta _2, \theta _4$, and ${\varvec{W}}$ is a matrix that adjusts weights of the joints. ${\varvec{\theta }}^{ref}$ are the joint angles given as a reference and ${\varvec{\theta }}$ is calculated as close to this value as possible.

The evaluation function is minimized with respect to $\theta _2, \theta _4$ as follows:

$$\begin{aligned} \mathop {\mathrm{arg~min}}\limits _{\theta _2, \theta _4} V(\theta _2, \theta _4, {\varvec{T}}_e, {\varvec{W}}, {\varvec{\theta }}^{ref}). \end{aligned}$$

(5)

To minimize this, firstly, the possible range of $\theta _2, \theta _4$ is calculated from the range of motion of the joints as simultaneous inequalities. A grid search is then performed to find possible values of $\theta _2, \theta _4$ and finally, the minimum value is calculated from (4) by Hooke-and-Jeeves method [35]. Using this method, the following has been achieved:

• This calculation is 40× faster than the numerical method.

• It has a 1.25× higher evaluation value (4) than the numerical method.

• It can solve 100% of a given potentially solvable IK problem set (compared to 85% by the numerical method).

Here we use the algorithm of [36] as the numerical method.

Motion execution

Time optimal path parameterization

The geometrical whole-body trajectory calculated by the motion planning section does not contain time information. Generally, in order to accurately move a robot including wheels in accordance with the geometrical whole body trajectory, it is necessary to consider the limits of the motors and the frictional forces. Since the HSR adopts a dual-wheel caster-drive mechanism, there is a relation to the limitation of the speed and torque of the wheel, and the speeds of translation and rotation change in a complicated way. Therefore, it is difficult to set a limiting value for the command of the speed. To solve this problem, we used the framework of time optimal path parameterization (TOPP) based on the numerical integration [37,38,39,40,41,42,43] as follows.

The dynamics system of a robot can be described as

$$\begin{aligned} {\varvec{\tau }} = {\varvec{M(q)\ddot{q}+ {\dot{q}}^{T}C(q)\dot{q}+D(q)\dot{q}+g(q)}} \end{aligned}$$

(6)

where ${\varvec{q}}\in R^n$ is configuration vector of the robot, ${\varvec{\tau }} \in R^n$ is the torque and force vector, ${\varvec{M,C,D}} \in R^{n \times n}$ and $g \in R^n$ is inertial matrix, centrifugal/Coriolis matrix viscosity matrix and gravity vector, respectively.

The constraints of ${\varvec{\tau }}$ is denoted by

$$\begin{aligned} {\varvec{{\tau }}}^{min} \le {\varvec{\tau }} \le {\varvec{{\tau }}}^{max} \end{aligned}$$

(7)

where ${\varvec{{\tau }}}^{min}$ and ${\varvec{\tau }}^{max}$ is the minimum and maximum torque.

The constraints of ${\varvec{\dot{q}}}$ is denoted by

$$\begin{aligned} |{{\varvec{\dot{q}}}}| \le {\varvec{\dot{q}}}^{max} \end{aligned}$$

(8)

where ${\varvec{{\dot{q}}}}^{max}$ is the maximum velocity.

Using one parameter of the path s, the target path is defined as

$$\begin{aligned} {\varvec{q}} = {\varvec{f}}(s) \end{aligned}$$

(9)

where $s=0$ is the start point and $s=s_{end} > 0$ is the end point of the path. The following equations are derived by differentiating (9).

$$\begin{aligned} {\varvec{\dot{q}}}= & {} {\varvec{f'}}(s)\dot{s} \end{aligned}$$

(10)

$$\begin{aligned} {\varvec{\ddot{q}}}= & {} {\varvec{f'}}(s)\ddot{s} + {\varvec{f''}}(s)\dot{s}^2 \end{aligned}$$

(11)

By substituting (6), (9), (10), (11), the constraints (7) and (8) could be parametrized by s as follows:

$$\begin{aligned} {\varvec{{\tau }}}^{min} \le {\varvec{a}}(s)\ddot{s}+{\varvec{b}}(s)\dot{s}^2+{\varvec{c}}(s)\dot{s}+{\varvec{d}}(s) \le {\varvec{{\tau }}}^{max} \end{aligned}$$

(12)

where ${\varvec{a}}(s), {\varvec{b}}(s), {\varvec{c}}(s), {\varvec{d}}(s)$ are coefficients calculated from s. The following condition is derived by (12).

$$\begin{aligned} {\ddot{s}^{min}(s,\dot{s})} \le \ddot{s} \le {\ddot{s}^{max}(s,\dot{s})} \end{aligned}$$

(13)

where ${\ddot{s}^{min}(s,\dot{s})}$, ${\ddot{s}^{max}(s,\dot{s})}$ is the minimum and maximum acceleration of s calculated by s and $\dot{s}$.

By substituting (10), the constraints (8) could be parametrized by s as follows:

$$\begin{aligned} |{{\varvec{f'}}(s)\dot{s}}| \le {\varvec{\dot{q}}}^{max} . \end{aligned}$$

(14)

The path and constraints are parameterized by s. A profile of $\dot{s}$ along the path can be obtained by the numerical integration from $(s,\dot{s})=(0,0)$ to $(s_{end},0)$. Figure 20 shows an example of a time optimal trajectory on the phase plane ($s-\dot{s}$ plane).

Fig. 20

$s-\dot{s}$ Plane representation of time optimal trajectory [43]

Based on Pontryagin’s maximum principle, the time optimal trajectory is calculated from bang-bang control using minimum or maximum acceleration inputs. The time optimal trajectory should be integrated with maximum acceleration $\ddot{s}^{max}(s,\dot{s})$ while acceleration and with minimum acceleration $\ddot{s}^{min}(s,\dot{s})$ while deceleration as in (13). The equation $\ddot{s}^{min}(s,\dot{s})$ = $\ddot{s}^{max}(s,\dot{s})$ draws an curve to show the boundary of the admissible and inadmissible region, called Maximum Velocity Curve (MVC). The integrated point cannot get over this curve, which means the torque constraints of all the joints are satisfied. The second condition is the velocity constraints. Even if the first condition is satisfied, $(s, \dot{s})$ should be restricted on the equation condition of (14). The boundary curve of (14) is called Velocity Limiting Curve (VLC). The integrated point cannot get over this curve, but should follow this curve which means a joint is moving at maximum speed. The time optimal path is calculated by searching “switching points” where $\ddot{s}$ switches to $\ddot{s}^{min}(s,\dot{s})$ or $\ddot{s}^{max}(s,\dot{s})$ in order to avoid entering inadmissible regions during the numerical integration.

Improvement of mobile base movement accuracy by laser odometry

When using the movement of the mobile base as part of the degrees of freedom for positioning the hand, it is necessary to ensure sufficient movement accuracy of the mobile base. Since dead reckoning using an encoder attached to the motor did not obtain sufficient accuracy, we adopted laser odometry [44] using LIDAR and measured the accuracy. When laser odometry is used, accuracy may be maintained even if slip exists between the wheel and the ground.

Results and discussion

Verify grasping capability

Here, we describe the evaluation results on the basic performance of grasping. We evaluated grasping objects chosen from 43 classes of [20]. Regarding the test conditions, we placed each object on a flat table with a height of 40 cm, and the robot gripped each object by remote control. However, magazine, book and thin objects such as mail, bill, credit, card and coin were evaluated in a stand-up position, because their flat position on the table made it difficult to be grasped properly. In order to confirm the stability of grasping, HSR moved over 5 mm step. At that time, we installed a 3-axis acceleration sensor in the hand and measured the acceleration. As a result, we confirmed that all objects could be stably gripped without falling or large deviation under the disturbance of the maximum acceleration of 33.15 m/s². The objects used in the experiments and the pictures of grasping by HSR are shown in Table 2. Note that in some cases, we substituted similar objects, for which we left comments in Object Class column of Table 2. In addition, we evaluated sucking the above 5 thin objects from the floor by remote control and confirmed that HSR was capable of sucking all of them.

Table 2 List of objects in [20] and pictures of grasping

Verify autonomous task capability

We developed promising tasks built around Fetch and Carry (bring an object as commanded by the user) and Tidy-Up as home mobile manipulator tasks, and tested the performance of the HSR. The study was conducted in a test environment that mimics a 2-bedroom apartment. Both furniture and environmental information were given in advance, AR markers were used for furniture recognition, and only the target object was recognized without the markers. Figure 21 shows an example of the Fetch and Carry task. This example was set up such that the robot received a voice instruction from the user, who was sitting on the bed in the bedroom. The instruction was to go take the drink in the plastic bottle from the shelves in the next room and hand it to the user. There was a closed door between the bedroom and the next room, and the robot was required to recognize the door and open it. Figure 22 shows the task of tidying up the floor and the task of tidying up the table. On the floor tidying-up task, there were an average of three trash items over the area of 1.5 × 1.5 m, and the goal was to put them in the basket at a designated place. The goal of cleaning up the table was to put all 6 household items on the dining table into the basket located next to the table. As a result of the experiment, the robot was able to autonomously complete all of the tasks described above. This confirmed that HSR possessed sufficient capacity to handle those tasks.

Fig. 21

Fetch and carry task [3]. a Open door, b Fetch, c Hand over

Fig. 22

Tidy-up task on floor and table [3]. a Floor, b Table

Validate usability as research platform

To check whether HSR could be enough for the research use, we held a hackathon, which is an event to compete collaborative computer programming within fixed days, in 2015. 8 teams consisting of 38 researchers and university students joined the event and competed original tasks of HSR. The event was only 3 days. However almost all the teams could show their final demonstrations successfully despite of such a short period of time (Fig. 23). After the event, we ask all the team members about the impression of HSR as the research platform (Fig. 24). As a result, majority of the team members were satisfied with HSR. Therefore we concluded that we could provide HSR to participants in robot competitions and various collaboration partners as the research platform.

Fig. 23

Examples of HSR hackathon demonstration. a Cooking task, b Dancing with HSR

Fig. 24

Results of questionnaire to participants

Validate effect at RoboCup 2017 and 2018

RoboCup@Home DSPL (Domestic Standard Platform League) was held at the RoboCup Nagoya Competition in 2017 and HSR was used as its Standard Platform (SP). Ten teams were selected for DSPL through a public offering and all of them accepted and participated. In order to verify the effect of introducing HSR as the SP, it was compared to the OPL (Open Platform League) competition, where custom robots were used instead. Whilst this was the first DSPL event, the OPL consisted of teams who had used their custom robots in at least two previous OPL events. Two competitions were selected for direct comparison between DSPL and OPL—General Purpose Service Robot (GPSR) and Storing Groceries. GPSR was a competition in which the robot was required to understand complicated and randomly generated commands and then perform tasks by understanding the voice commands; therefore, it was highly dependent on language understanding software specialized for GPSR. The Storing Groceries task consisted of grasping and moving everyday items, and the challenge was dependent on the basic performance of the mobile manipulator. In order to mitigate the effects of teams who simply may have not prepared sufficiently, the total score from the top three teams was tallied for each competition. As shown in Fig. 25, the OPL achieved a higher aggregate score for GPSR. This is believed to be due to the prior experience of those teams in participating in RoboCup.

Fig. 25

(Obtained from [48])

Total points of top three teams for GPSR in 2017

Fig. 26

(Obtained from [48])

Total points of top three teams for Storing Groceries in 2017

On the other hand, DSPL achieved a higher score in Storing Groceries despite their first-time participation, as shown in Fig. 26.

Figure 27 shows the actual Storing Groceries scenes of Team Hibikino-Musashi@Home [45], which participated both OPL and DSPL in RoboCup Nagoya. OPL and DSPL were operated under the same rule, although it should be noted that we could not compare OPL and DSPL under the completely identical conditions, because referees rearrange objects in each turn. However, both OPL and DSPL use the same room, same furniture, same objects and similar object layout as shown in Fig 27. Therefore we concluded Fig. 26 could represent the performance reliable to a certain extent.

Fig. 27

Storing Groceries by Hibikino-Musashi@Home in Nagoya 2017. a OPL, b DSPL

In 2018, RoboCup was held in Montreal. As shown in Fig. 28, total points for GPSR of DSPL increased. The increase in the number of experiences of the DSPL teams is estimated to be the reason. In addition, the result of Storing Groceries in Fig. 29 shows that DSPL was still better than OPL. These results suggest that HSR has potential performance as a home mobile manipulator.

Fig. 28

(Obtained from [49])

Total points of top three teams for GPSR in 2018

Fig. 29

(Obtained from [49])

Total points of top three teams for Storing Groceries in 2018

Conclusion

In this paper, we described HSR’s development history, technical details, experimental results. We expect that this paper will provide useful information for current and prospective researchers.

The design of HSR is still under active development and we will improve it by continually reflecting user requests. Together with research activities, we are carrying out various field experiments in actual environments (Fig. 30). We believe that the cycle of researches and field experiments is essential to realize the domestic mobile manipulator. To achieve it, we will work with the HSR Developers Community.

Fig. 30

Examples of field experiments. a Test in nursing facility, b Test in real house [3]