This paper presents a study of an audiovisual interface-based drumming system for multimodal human–robot interaction. The interactive multimodal drumming game is used in conjunction with humanoid robots to establish an audiovisual interactive interface. This study is part of a project to design robot and avatar assistants for education and therapy, especially for children with special needs. It specifically focuses on evaluating robot/virtual avatar tutors, tangible interaction devices, and mobile multimedia devices within a simple drumming-based interactive music tutoring game scenario. Several parameters, including the effect of the embodiment of the tutor/interface and the presence of feedback and training mechanisms, were the focus of interest. For that purpose, we created an interactive drumming game relying on turn-taking and imitation principles, in which a human user is able to play drums with a humanoid robot (Robotic Drum Mate). Three interactive scenarios with different experimental setups for humanoid robots and mobile devices were developed and tested. As a part of those scenarios, a system that enables drum strokes to be automatically detected and recognized using auditory cues was implemented and incorporated into the experimental framework. We verified the applicability and effectiveness of the proposed system in a drum-playing game with adult human test subjects by evaluating it both objectively and subjectively. The results showed that the physical robot tutor, the feedback and training mechanisms had a positive effect on the subjects’ performance and, as expected, although the physical medium is preferred, the virtual medium for drumming caused less failure.

本文介绍了一种用于多模式人机交互的基于视听界面的鼓系统的研究。交互式多模式击鼓游戏与人形机器人一起使用，以建立视听交互界面。这项研究是设计用于教育和治疗（尤其是有特殊需求的儿童）的机器人和化身助手的项目的一部分。它特别着重于评估基于简单的基于鼓的互动音乐辅导游戏场景中的机器人/虚拟化身导师，有形的交互设备和移动多媒体设备。几个参数，包括导师/界面的实施方式的效果以及反馈和培训机制的存在，是关注的焦点。为了这个目的，我们创建了一个基于转弯和模仿原理的交互式击鼓游戏，其中人类用户可以使用类人机器人（Robotic Drum Mate）来演奏鼓。针对人形机器人和移动设备，开发了三种具有不同实验设置的交互方案。作为这些方案的一部分，实施了一个系统，该系统能够使用听觉提示自动检测和识别鼓声，并将其并入实验框架。我们通过客观和主观的评估，验证了该系统在具有成人人体测试对象的鼓类游戏中的适用性和有效性。结果表明，物理机器人导师，反馈和培训机制对受试者的表现有积极影响，并且如预期的那样，