We present research for automatic assessment of pianist hand posture that is intended to help beginning piano students improve their piano-playing technique during practice sessions. To automatically assess a student's hand posture, we propose a system that is able to recognize three categories of postures from a single depth map containing a pianist's hands during performance. This is achieved through a computer vision pipeline that uses machine learning on the depth maps for both hand segmentation and detection of hand posture. First, we segment the left and right hands from the scene captured in the depth map using per-pixel classification. To train the hand-segmentation models, we experiment with two feature descriptors, depth image features and depth context features, that describe the context of individual pixels' neighborhoods. After the hands have been segmented from the depth map, a posture-detection model classifies each hand as one of three possible posture categories: correct posture, low wrists, or flat hands. Two methods are tested for extracting descriptors from the segmented hands, histograms of oriented gradients and histograms of normal vectors. To account for variation in hand size and practice space, detection models are individually built for each student using support vector machines with the extracted descriptors. We validate this approach using a data set that was collected by recording four beginning piano students while performing standard practice exercises. The results presented in this article show the effectiveness of this approach, with depth context features and histograms of normal vectors performing the best.

中文翻译：

---

使用深度数据检测钢琴演奏中的手部姿势

我们提出了自动评估钢琴家手部姿势的研究，旨在帮助钢琴初学者在练习期间提高他们的钢琴演奏技巧。为了自动评估学生的手部姿势，我们提出了一种系统，该系统能够在演奏期间从包含钢琴家手部的单个深度图中识别三类姿势。这是通过计算机视觉管道实现的，该管道在深度图上使用机器学习进行手部分割和手部姿势检测。首先，我们使用逐像素分类从深度图中捕获的场景中分割左手和右手。为了训练手动分割模型，我们试验了两个特征描述符，深度图像特征和深度上下文特征，它们描述了单个像素邻域的上下文。从深度图中分割双手后，姿势检测模型将每只手分类为三种可能的姿势类别之一：正确姿势、低手腕或平手。测试了两种方法来从分割的手中提取描述符，定向梯度的直方图和法向量的直方图。为了考虑手的大小和练习空间的变化，使用支持向量机和提取的描述符为每个学生单独构建检测模型。我们使用一个数据集来验证这种方法，该数据集是通过在执行标准练习练习时记录四名钢琴初学者来收集的。本文中展示的结果显示了这种方法的有效性，其中深度上下文特征和法向量的直方图表现最佳。