By looking at a person’s hands, one can often tell what the person is going to do next, how his/her hands are moving and where they will be, because an actor’s intentions shape his/her movement kinematics during action execution. Similarly, active systems with real-time constraints must not simply rely on passive video-segment classification, but they have to continuously update their estimates and predict future actions. In this paper, we study the prediction of dexterous actions. We recorded videos of subjects performing different manipulation actions on the same object, such as “squeezing”, “flipping”, “washing”, “wiping” and “scratching” with a sponge. In psychophysical experiments, we evaluated human observers’ skills in predicting actions from video sequences of different length, depicting the hand movement in the preparation and execution of actions before and after contact with the object. We then developed a recurrent neural network based method for action prediction using as input image patches around the hand. We also used the same formalism to predict the forces on the finger tips using for training synchronized video and force data streams. Evaluations on two new datasets show that our system closely matches human performance in the recognition task, and demonstrate the ability of our algorithms to predict in real time what and how a dexterous action is performed.

中文翻译：

---

操纵动作的预测

通过观察一个人的手，人们通常可以知道这个人接下来要做什么，他/她的手如何移动以及它们将在哪里，因为在动作执行过程中，演员的意图塑造了他/她的运动动力学。同样，具有实时约束的主动系统不能简单地依赖被动视频片段分类，而是必须不断更新估计并预测未来的动作。在本文中，我们研究了灵巧动作的预测。我们记录了受试者对同一物体执行不同操作动作的视频，例如用海绵“挤压”、“翻转”、“洗涤”、“擦拭”和“抓挠”。在心理物理学实验中，我们评估了人类观察者从不同长度的视频序列中预测动作的技能，描绘了在与物体接触之前和之后的动作准备和执行过程中的手部运动。然后，我们开发了一种基于循环神经网络的动作预测方法，使用手周围的输入图像块。我们还使用相同的形式来预测指尖上的力，用于训练同步视频和力数据流。对两个新数据集的评估表明，我们的系统在识别任务中与人类的表现非常匹配，并证明了我们的算法能够实时预测执行灵巧动作的内容和方式。我们还使用相同的形式来预测指尖上的力，用于训练同步视频和力数据流。对两个新数据集的评估表明，我们的系统在识别任务中与人类的表现非常匹配，并证明了我们的算法能够实时预测执行灵巧动作的内容和方式。我们还使用相同的形式来预测指尖上的力，用于训练同步视频和力数据流。对两个新数据集的评估表明，我们的系统在识别任务中与人类的表现非常匹配，并证明了我们的算法能够实时预测执行灵巧动作的内容和方式。