Upper limb and hand functionality is critical to many activities of daily living, and the amputation of one can lead to significant functionality loss for individuals. From this perspective, advanced prosthetic hands of the future are anticipated to benefit from improved shared control between a robotic hand and its human user, but more importantly from the improved capability to infer human intent from multimodal sensor data to provide the robotic hand perception abilities regarding the operational context. Such multimodal sensor data may include various environment sensors including vision, as well as human physiology and behavior sensors including electromyography and inertial measurement units. A fusion methodology for environmental state and human intent estimation can combine these sources of evidence in order to help prosthetic hand motion planning and control. In this paper, we present a dataset of this type that was gathered with the anticipation of cameras being built into prosthetic hands, and computer vision methods will need to assess this hand-view visual evidence in order to estimate human intent. Specifically, paired images from human eye-view and hand-view of various objects placed at different orientations have been captured at the initial state of grasping trials, followed by paired video, EMG and IMU from the arm of the human during a grasp, lift, put-down, and retract style trial structure. For each trial, based on eye-view images of the scene showing the hand and object on a table, multiple humans were asked to sort in decreasing order of preference, five grasp types appropriate for the object in its given configuration relative to the hand. The potential utility of paired eye-view and hand-view images was illustrated by training a convolutional neural network to process hand-view images in order to predict eye-view labels assigned by humans.

中文翻译：

---

HANDS：一个多模式数据集，用于对假肢中的人类抓握意图进行建模

上肢和手的功能对于日常生活中的许多活动至关重要，而截肢可能会导致个人的重大功能丧失。从这个角度来看，预计未来的先进假肢将受益于改进的机械手及其人类用户之间的共享控制，但更重要的是，其改进的功能可以从多模式传感器数据中推断出人类意图，从而提供有关以下方面的机械手感知能力：操作环境。这样的多模式传感器数据可以包括各种环境传感器，包括视觉，以及人类生理和行为传感器，包括肌电图和惯性测量单元。用于环境状态和人类意图估计的融合方法可以结合这些证据来源，以帮助进行假肢手部运动的计划和控制。在本文中，我们提出了这种类型的数据集，该数据集是在假手中内置摄像头的预期下收集的，而计算机视觉方法将需要评估此手视视觉证据以估计人的意图。具体而言，在抓握试验的初始状态下，已捕获了来自人眼和手的不同方向放置的各种物体的配对图像，然后在抓握，抬起过程中从人的手臂捕获了成对的视频，EMG和IMU ，放下和收回样式的试用结构。对于每次试验，根据场景的眼睛图像在桌子上显示手和物体，要求多个人按优先级从高到低的顺序进行排序，五种抓握类型适用于相对于手的给定配置中的对象。通过训练卷积神经网络来处理手视图像以预测人为分配的眼视标签，说明了成对的眼视和手视图像的潜在效用。