This paper presents an ongoing project focused on the framework Listen by Looking, that includes several tools to support the implementation of digital games, allowing the interactive display of music sheets, piano rolls and dialog windows with contextual information. Using this framework, a serious game for mobile devices was designed to entertain and teach music to users by listening to songs and answering questions about the melodies and musical structures learned during the game. The usefulness of game was assessed in a controlled setup, by comparing the answers of the app users with those of people who listened the same music without the app. Furthermore, an external synchronization tool through a data-over-sound channel has been tested in order to simultaneously manage a wide network of devices. The results are encouraging for future developments of the framework, with the possibility of using it during a public performance and with different games.

本文提出了一个正在进行的项目，该项目侧重于“边听边看”框架，其中包括支持数字游戏实施的多种工具，允许交互式显示乐谱，钢琴卷和带有上下文信息的对话窗口。使用此框架，针对移动设备设计的一款严肃的游戏旨在通过听歌并回答有关在游戏中学习的旋律和音乐结构的问题来娱乐和向用户传授音乐。通过将应用程序用户的答案与未使用该应用程序听相同音乐的人的答案进行比较，以受控设置评估了游戏的有用性。此外，已经测试了通过声音数据通道的外部同步工具，以便同时管理广泛的设备网络。结果对于该框架的未来发展是令人鼓舞的，