1 Introduction

Increasingly, there is a need for industrial robots that can manufacture or assemble products which are bespoke or have a variable design and bill of materials [1]. Application areas include aerospace, the space industry, and industrial assembly [2, 3], where it is useful to manufacture small runs of specific assemblies and rapidly adapt to changing or evolving designs. Developing adaptive manufacturing systems has the potential to reduce waste, increase the rate of assembly, and allow bespoke design.

There has been much existing work in this area, in particular with a focus on how dual-arm systems can work collaboratively with humans to assemble systems [4,5,6,7,8]. It is now necessary to advance this research to move towards fully autonomous, flexible, and adaptive assembly systems.

It has also been shown that robotics competitions provide a means of driving innovation, comparing and benchmarking different technologies, and quantifying the quality and applicability of research [9]. The DARPA robot challenge demonstrated how the limitations of existing robot systems could be identified, and how overtime solutions can be developed which ‘solve’ the challenges posed, [10]. There has been a recent increase in the number of manipulation-based robotics competitions which seek to address mobile manipulation, service manipulation tasks, soft robotic manipulation, and warehouse picking [11,12,13,14]. Robot competitions provide a great opportunity to drive and test robotic research into industrial manipulation. It provides an opportunity to develop agile, efficient, and lean assembly systems which enable the production of bespoke assemblies [15].

The World Robot Summit (WRS) Industrial Assembly Challenge is one such robot competition in the area of industrial robotics [16]. The competition has an overall aim of developing robotic technologies, vision and learning approaches to allow the assembly of complex systems, and how the system can respond to assembling a varied product. This is both a challenging research question and also a novel competition framework. The major focus is on the ability to perform flexible manipulation opposed to showing high performance of a single product. The challenge is summarised in Fig. 1.

We propose an approach to achieve adaptive manufacturing where general-purpose mechanical grippers and end effectors are developed to allow a wide range of tools and different objects to be manipulated. The mechanical approaches also seek to minimise the precision or accuracy required, by utilising material properties and software-based control procedures to reduce the requirement for high accuracy and precision. This enables unknown and previously unseen parts to be easily manipulated. Additionally, we have developed modular and scalable vision and learning approaches which allow rapid detection of previously unknown or unseen parts.

In this paper, the exact competition problem definition and system overview are given in Sect. 2. Following this, Sect. 3 introduces the novel techniques which have been implemented to achieve adaptive assembly and manufacturing. Section 4 then presents the results of experimental tests, with the paper finishing with a discussion and conclusion on the system developed and also the role of competitions in driving and testing research in this area.

2 Problem definition and system overview

2.1 World robot summit assembly challenge

The World Robot Summit [17] is an international Robotics Competition which was held for the first time in October 2018. There were a number of leagues, notably rescue, service, and assembly. The industrial assembly task is the focus of this work. The overall aim is to develop adaptive industrial robotic assembly systems. Specifically to develop a robot system which could manufacture a belt drive system, in an adaptive way such that if changes are made to the design of the belt system, minimal changes are required from the robot in terms of both software and hardware. The belt drive system which must be assembled is shown in Fig. 1. The specific tasks in the competition include:

Fig. 1
figure 1

The four tasks which make up the WRS Industrial Assembly challenge: task board, kitting, assembly, and adaptive assembly

Full size image

  • Task board This is to demonstrate the key components which must then be integrated to achieve full assembly of the belt drive system. Each of the individual parts is placed on a task mat. The specific parts must be identified, classified, and then manipulated and placed on the task board, which replicates the physical task required in the full assembly challenge. For example, screws must be picked up, manipulated and screwed into the task board, and shafts inserted into holes.

  • Kitting Given a bill of materials, the correct parts must be picked from parts bins where the parts are arranged randomly. This involves picking small items (e.g. washers, nuts, bolts), larger items (e.g. motors, pulleys, shafts), and also flexible items (the pulley belt). The objects must also be placed into kitting trays, for example, bolts must be placed with accuracy into screw holders.

  • Assembly For a known set of parts and assembly details, a belt drive system must be assembled autonomously (including pick up the parts). The main parts which form the belt assembly include a motor, output pulley, shaft housing, shaft, end cap, large pulley, and belt. This must be completed to both a high standard and also as quickly as possible.

  • Surprise assembly Given an altered assembly, for example, with different sized parts or an assembly with a different structure, the updated belt drive should be assembled. Only one hour preparation time is given with the new parts to allow development and testing. Thus, the robot should be designed to be adaptive so only minimal changes are required in the mechanics and software to achieve this assembly challenge.

Fig. 2
figure 2

Overview block diagram of the system showing the vision, control, and mechanical parts of the system

Full size image
Fig. 3
figure 3

Pictures showing the overall system setup with the two arms (UR5, Universal Robots), overhead camera, lighting, and three custom grippers

Full size image

In this paper, we focus on the three most challenging aspects: kitting, assembly, and the most interesting aspect from a research perspective, flexible assembly.

2.2 System overview

The system developed uses two 6 axis robotic arms (UR5, Universal Robots) working collaboratively together. Custom arm control and motion planning software has been developed. Three custom end-effectors have been developed which are controlled via micro-controllers. A single 4K Ultra HD Camera (BRIO, Logitech) is used above for vision, and the internal force feedback from the arms is used to enable complex multi-arm movements. A block diagram of the system is shown in Fig. 2, with the entire setup as shown in Fig. 3.

The custom grippers are mounted at the end of the UR5 arms. These are controlled by the micro-controllers which communicate with the main computer over serial. The pincer gripper uses a DC motor and threaded rod mechanism to move two fingers to provide high-force pinching gripping to pick up objects of different sizes. The micro-controller uses current feedback from the DC motor to control the position and detect when the gripper closes.

The second gripper, a custom rotating gripper, has a variable size aperture which also allows infinite rotation. This has been designed to allow for grasping and manipulation of a wide variety of different objects of different sizes and shapes with a minimal control requirement.

The final gripper, the tack gripper, uses sticky Blu-tack to grasp small parts for the kitting challenge. A servo is then controlled by the micro-controller to remove the item. This allows for robust, high-speed pick and place of many varied parts with very minimal control and with a low reliance on precision.

This research aims to investigate how agile and adaptive manufacturing can be achieved by:

  • Reducing the control complexity by developing mechanical systems which are flexible to changes in locations or parts and size and shape of parts

  • Achieving agility through general-purpose grippers and software which allow a wide range of objects to be manipulated.

  • Developing robotic manipulators can use a wide range of tools of different sizes, which all use the same underlying control approaches.

  • Using a kitting approach which is general for all parts, requiring low precision and enables the same control and manipulation approach to be used for all

  • Flexibility and adaptability through modular software and vision system

Fig. 4
figure 4

The custom grippers developed for flexible assembly. a rotary gripper, b two-finger pincher gripper, and c kitting gripper

Full size image

3 Adaptive manufacturing methods

3.1 Adaptive assembly grippers

The first gripper, the pincher gripper (Fig. 4b), is a simple parallel plate mechanism with reinforced 3d printed fingers. The fingers are shaped and elongated to pick in a variety of parts including motors, fixing plates, and smaller parts such as bolts. This gripper uses an Mbed micro-controller to communicate with the PC and to control the single motor and read the motor current and switch. The motor current defines the end-stops and the force applied to any object in the gripper. The switch defines the home position, and this central position saves time when opening and closing.

The second gripper, the rotary gripper (Fig. 4a), is a more specialised design. This gripper design uses two motors, one to open and close the fingers and one to rotate the fingers. The fingers can rotate continuously and independently of the position, enabling the screwing in motion for assembly. The fingers contain springs which force the grippers open; this is exploited to enable another method of picking by gripping from the inside of a part. When the position bearing is extended, the fingers are squeezed together. This manipulator excels at picking up cylinders and triangular or hexagonal prisms. The limits of the open and closed position of the gripper are detected by monitoring the current in the motor. The rotations of the fingers are tracked by a single micro-switch; this also allows position calibration of the rotating head. The gripper enables the robot to adapt, by picking tools of different heads and sizes, and perform agile assembly with new parts.

The key task which this gripper enables is the use of tools, specifically the use of Allen keys to allow screwing of bolts. We have developed an innovative approach to enable bolts to stay at the end of Allen keys. This is the use of grease. The Allen key is dipped in grease and then inserted into bolt heads. When lifted, the screw remains attached to the bolt. This combination of the rotary gripper and tool usage makes this gripper very powerful and highly adaptive to many different tool types.

3.2 Adaptive kitting grippers

One fixed size gripper has been designed which allows kitting of all of the small parts (washers, nuts, bolts, etc.). This gripper has a soft adhesive pad (made from Blu-tack) which allows parts to be picked using adhesion. Adhesion has been previously shown to be an effective method for pick and place [18] and also for achieving climbing or holding onto walls [19].

To remove the part from the pad, a servo-controlled sleeve can push the part of the adhesive pad. The size of the adhesive pad has been designed to have sufficient adhesive and tack force to lift a single piece of all the small parts whilst also only allow picking of one piece to minimise the precision required from the vision system. Figure 4c shows the gripper developed. This method achieves adaptive gripping as the same gripper can be used to grip many parts of different form factors with no physical changes required. It also requires minimal accuracy and precision from the vision to minimise the development of custom systems.

3.3 Collaborative arm control

Collaborative two-arm control is required to achieve some of the complex assembly tasks. This provides the ability to:

  • Pass parts between grippers so they can be held in an optimum position or by an alternative gripper.

  • Hold certain parts stable whilst another gripper screws or otherwise interfaces with other parts.

  • Perform sub-assemblies, for example, put washers on a screw held in another hand before this sub-assembly can then be integrated with the rest of the system.

To achieve this, the two arms were calibrated together by determining the co-ordinates for the two robots at three separate points. One point is the base point, the other two points form vectors which are used to extrapolate the positions of the robots relative to each other. This allows the two hands to move together or move relative to each other. Figure 5 shows collaborative two-arm and gripper control which enables complex movements including the assembly and bolting of a motor to a motor bracket.

Fig. 5
figure 5

Two-hand manipulation to enable screwing of a motor into the motor plates

Full size image

3.4 Force feedback

Force feedback algorithms for insertion and hole finding have been developed to remove the need for high precision and hard coding. The reduction of the need for hard coding makes the system more adaptive when the parts which must be assembled are changed. The force measurements from the UR5 are imprecise and fluctuate especially when accelerating. This is overcome by monitoring the difference in forces and thresholding. The hole finding function, Fig. 6, attempts to find the hole by moving towards it until a force limit has been exceeded or the final position has been reached. If the final position has not been reached, or no drop in force is detected, this indicates the hole has not been found. Thus, the end-effector performs a hunt, moving in circles of increasing radii until the force drops significantly, indicating the hole has been located. The drop in force indicates that the hole has been found; the robot then attempts a final force move to fully insert into the hole.

Fig. 6
figure 6

Flowchart for basic hole finding function using force feedback

Full size image

This insert function has many different input parameters which can be tuned for different holes and environments, including the force limit, circle radii, and speed of rotation and whether the hunting continues if the final force move in Fig. 6 does not reach the final position.

This insert function is used widely within the assembly. It is used to insert Allen keys into bolt head, the pulley onto the motor shaft, bolts into bolt holes, the shaft into the shaft housing and much more. The universal nature of the function allows it to be used for different parts, different locations, and to serve different overall functions.

3.5 Vision

Vision is used to detect and localise the different parts. It was important to develop a vision and learning system which can be rapidly expanded to include new or altered parts.

3.6 Data set and image pre-processing

A core part of the object recognition process is image pre-processing. The data set used for the experiments comprises 1500 RGB images of dimension \(1920\times 1080\times 3\). Each image was taken by a camera directly above the workspace, facing perpendicularly downwards on the object mat (see Fig. 7a). For each image in the data set, the mat was shifted and rotated manually at random; moreover, the objects are relocated in different positions. The labelling of the data set is achieved by selecting a bounding box over each object on the original images and a method which decreases labelling times.

Fig. 7
figure 7

Key elements of the vision pipeline showing the vision work-space and the pre-processed example images used in the identification stage

Full size image

Before feeding the data to a network, it is necessary to make the objects in the data set comparable to each other. First, we convert the RGB images into grey scales, reducing the dimensionality of the input layer and thus training time, and forcing the network to focus on geometrical information, rather than colour discrimination. We choose an object-detection image size of \(300\times 300\), based on the dimension of the largest object in the figures, i.e. the belt, and automatically crop each object based on its labelled bounding box. Each object image is padded on each side, to reach the \(300\times 300\) standard dimension. As shown in Fig. 7b, the object will thus be at the centre of the image, surrounded by ‘0’ pixel values. Here, the dimension of the outer padding provides useful information for object discrimination. Moreover, ‘0’ pixels do not excite any weight units in the network and will thus be naturally discarded for discrimination. After the pre-processing procedure, the data set corresponds to 15,200 images, each containing a single object (Fig. 8).

3.7 Inception convolutional neural network for object recognition

To perform object recognition and to cope with objects of different sizes, we devised a shallow inception convolutional neural network. The ICNN devised is comprised of an input layer, an inception layer, two convolutional layers with 3 \(\times \) 3 kernels with 32 channels (shallow convolutions), and two fully connected layers before an output layer classifying each object into its class with a softmax function. The inception layer is formed of 4 parallel convolution layers with 20 channels and increasingly larger kernels of 2 \(\times \) 2, 3 \(\times \) 3, 5 \(\times \) 5 and 7 \(\times \) 7. The relative size of kernels allows the network to learn local features at different scales, thus coping with the varying size of the objects in the data set. All units in the network perform a ReLu non-linear transformation.

Fig. 8
figure 8

The network design for the object recognition in flexible assembly

Full size image

4 Experimental results and demonstration

In this section, we include experimental results of the system and also demonstration of the performance and abilities of the system.

4.1 Vision

4.2 Object recognition

We use TensorFlow [20] to create the network described in Sect. 3.7 and train the network with 75% of the images in the data set, each pre-processed, padded, and containing a single object at its centre. We use RMSProp [21] with \(decay=0.9\), \(\epsilon =1e-10\), \(learning\ rate=0.0001\), and no momentum and perform gradient descent on the softmax cross entropy between the logits in output by the neural network and the one-hot encoding of the object labels.

Table 1 Results from the kitting experiments showing the success of picking the different items, success when removing and also the average number of items picked
Full size table

Similarly, we devise a new convolutional neural network, identical to the network designed in Sect. 3.7, but substitute the inception layer devised by a 5 \(\times \) 5 convolution layer with 80 channels. We train both networks to compare their performance and early stop only when the validation error, computed over 25% of the data set, does not improve for over 10 epochs. Figures 9 and 10 show the validation error and accuracy of both networks over all epochs before early stopping. As clear from the figure, the ICNN performs overall better than its convolutional counterpart. Finally, the classic convolutional neural network reaches a maximum accuracy level of 78.33% and minimum error of 0.8873, in comparison to the ICNN, which reaches a max validation accuracy of 89.5 and minimum error of 0.6198, thus improving performance on both accounts.

Fig. 9
figure 9

Comparative validation error of the CNN and ICNN networks

Full size image
Fig. 10
figure 10

Comparative validation accuracy of the CNN and ICNN networks

Full size image

This vision system was used in the task board and kitting task, to accurately detect the parts and the location.

4.3 Kitting

To test the kitting system, the reliability of the system to pick a single item and place it in a kitting tray tested for the different items. The vision system was used to identify a specific part to pick. For circular items (washers, nut, spacers, etc.), the edge of the circular part was chosen as the grasping point. For other parts (bolts), the head was chosen as this provides the greatest contact area. The Blu-tack gripper was moved above the part and then lowered until the force feedback indicates that the gripper had made contact with the part (a minimum force threshold was met.) The results can be found in Table 1.

The gripper was highly successful with the larger parts (e.g. nut and end cap) where the parts are larger than the gripper and have a large surface area. Although washers could be reliably picked, often more than one could be picked as the washers are smaller in diameter than the diameter of the soft adhesive; additionally, in some cases, the release mechanism did not remove the item as the item is smaller than the size of the adhesive unit. Pictures showing the gripping of these items are shown in Fig. 11.

Fig. 11
figure 11

Pictures showing the successful picking of a wide variety of different parts as required for the kitting challenge

Full size image

This approach provides a universal method to kitting and enables simple control strategies to be used to achieve a complex and challenging manipulation procedure.

4.4 Assembly manipulation tasks

Adaptive manipulation is achieved by using the grippers which show a huge range of abilities to pick and place and perform various manipulation of different parts and tools. The abilities to achieve these tasks in a variety of scenarios are shown in Fig. 12. The ability to pick up parts of highly varying form factor (diameter and height) is shown in Fig. 12a–c where a large pulley, thin shaft, and washer are all picked using the same gripper and using the same control function where the gripper is closed until the current feedback of motor rises above an appropriate threshold. To pick up parts where there is an inner hole or lip, the gripper can be closed tight lowered into the hold and then the lips of the gripper open, so the part can be gripped from the inside. This is shown in Fig. 12d where is possible to pick up the large and troublesome shaft housing using the inner lips of the gripper. Finally, it is also possible to pick up tools and Allen keys of a range of sizes. This is shown in Fig. 12e–g.

The adaptive nature can be quantified. It can pick up parts with a diameter or maximum diameter of 40 mm and likewise parts can be picked up using an inner hold with a minimum diameter of 30 mm. This includes the holding of prism-shaped tools (e.g. Allen keys/sockets) which are within these size limits.

Fig. 12
figure 12

Rotary gripper picking examples. a Pulley, b Shaft, c Washer, d Shaft housing, e M4 Allen key, f M2 Allen key, and g M10 Socket

Full size image
Fig. 13
figure 13

Image of the constructed pulley system (top) and the assembly with the surprise parts which were given only once before the start of competition and was achieved in around 20 min of competition time

Full size image

4.5 Adaptive manufacturing

The abilities of the system to perform adaptive, flexible manufacturing were tested in the final round of the WRS competition when it was necessary for the robot system to manufacture a belt drive system which has some changes in design which were not seen until 1 hour before the competition.

The original assembly (Fig. 13 top) and the adapted system which was manufactured during the competition are shown in Fig. 13. With only 1 hour practise time with these new updated parts, and then only 20 minutes of competition time, the assembly shown was manufactured. This included an assembly of a sprocket on the output shaft of the motor and also an M3 screw to hold this onto the motor shaft. Additionally, a new shaft and spacer were inserted into the bearing housing. Although additional sub-assembly tasks were developed, this was not achieved during the time allotted in the competition.

4.6 Competition results

The performance of the system was tested and benchmarked against other systems at the WRS competition. We came 2nd in the Kitting challenge and 2nd in the combined assembly and flexible assembly challenges. Indeed, we were the only team to demonstrate significant assembly in the surprise tasks. We were awarded the special innovation award, due to the abilities of the system to achieve flexible assembly and also the innovative and novel gripper designs.

5 Discussion

5.1 Adaptive manufacture

This research has demonstrated how adaptive manufacturing systems can be developed. This has been achieved by developing adaptive grippers which can pick up a wide range of parts and tools, and also accompanying control strategies which use force feedback to achieve resilience to changes in design. We have demonstrated the ability of the system by demonstrating how it can be used to assemble an updated assembly with very minimal changes required to the software and control.

To achieve flexibility in kitting, adhesive material has been used to develop a ‘Blu-tack’ gripper. This method of picking parts provides an approach to many parts, with the controlled uniform across all. This provides a system which can be rapidly changed to perform kitting of previously unseen parts.

This system has been tested in a competition environment, where, unlike a lab environment, the success could only be tested in a single run where it must ‘work’. There are no opportunities to rerun or retest, and it is a great benchmark to test the true ‘here and now’ capabilities of the system. However, further exploration is required to better understand the scope of the flexible assembly behaviour, to understand the extent to which these general-purpose grippers can be used, and to quantify the scope of this approach.

5.2 Role of competitions

This work also highlighted how competition can be used to drive innovation in industrial assembly. By developing a competition in this area where there is an unseen and unknown task, in this case the surprise/flexible assembly task, it was possible to truly test research of agile and flexible assemble. This competition identified how challenging this adaptive assembly task is. Performing automated assembly of the belt drive was sufficiently challenging. Further, developing this to achieve adaptive manufacturing was highly complex. The competition identified the remaining research which must be addressed to solve this problem.

This competition provided an opportunity to benchmark different research solutions and approaches. It also highlighted both the need and requirements from industry for flexible and adaptive robotic assembly systems and identified the research which is required to meet this goal. The inclusion of the flexible assembly task in the challenge forced teams to design and develop a robotic system which was adaptive and flexible opposed to hard-coding and engineering a specific solution to achieve a single task. The design of the competition in this way was shifted the research direction to address this more interesting and challenging and problem. This is an approach which should be used in further competitions such that competitions can be used to drive research innovation. In this way, the research does not become too closely defined by the specific aims of the competition and innovation and creativity is not stifled.