This study aims at sensing and understanding the worker’s activity in a human-centered intelligent manufacturing system. We propose a novel multi-modal approach for worker activity recognition by leveraging information from different sensors and in different modalities. Specifically, a smart armband and a visual camera are applied to capture Inertial Measurement Unit (IMU) signals and videos, respectively. For the IMU signals, we design two novel feature transform mechanisms, in both frequency and spatial domains, to assemble the captured IMU signals as images, which allow using convolutional neural networks to learn the most discriminative features. Along with the above two modalities, we propose two other modalities for the video data, i.e., at the video frame and video clip levels. Each of the four modalities returns a probability distribution on activity prediction. Then, these probability distributions are fused to output the worker activity classification result. A worker activity dataset is established, which at present contains 6 common activities in assembly tasks, i.e., grab a tool/part, hammer a nail, use a power-screwdriver, rest arms, turn a screwdriver, and use a wrench. The developed multi-modal approach is evaluated on this dataset and achieves recognition accuracies as high as 97% and 100% in the leave-one-out and half-half experiments, respectively.

这项研究旨在感知和理解以人为中心的智能制造系统中工人的活动。我们提出了一种新颖的多模式方法，通过利用来自不同传感器和不同模式的信息来识别工人的活动。具体而言，智能臂章和可视摄像机分别用于捕获惯性测量单元（IMU）信号和视频。对于IMU信号，我们在频域和空间域中设计了两种新颖的特征变换机制，以将捕获的IMU信号组装为图像，从而允许使用卷积神经网络来学习最有区别的特征。除了上述两种方式，我们还为视频数据提出了两种其他方式，即在视频帧和视频剪辑级别。四种模态中的每一种都返回活动预测的概率分布。然后，将这些概率分布融合以输出工人活动分类结果。建立了一个工人活动数据集，该数据集目前包含装配任务中的6个常见活动，即，抓住工具/零件，钉子，使用电动螺丝刀，固定臂，转动螺丝刀和使用扳手。在该数据集上评估了已开发的多模式方法，并在留一法实验和半半实验中分别实现了高达97％和100％的识别精度。锤打钉子，使用电动螺丝刀，放下手臂，转动螺丝刀，然后再使用扳手。在此数据集上评估了已开发的多模式方法，并在留一法实验和半半实验中分别实现了高达97％和100％的识别精度。锤打钉子，使用电动螺丝刀，放下手臂，转动螺丝刀，然后再使用扳手。在此数据集上评估了已开发的多模式方法，并在留一法实验和半半实验中分别实现了高达97％和100％的识别精度。