Automatic analysis of pronunciations for children with speech sound disorders

Highlights

• Propose two approaches for evaluating pronunciations for children with speechsound disorders.

• GOP-Algorithm – an algorithm based approach.

• GOP-SVM – a statistical learning based approach.

• Prove that the proposed approaches outperform the current GOP state-of-the art approach using several different tests.

• Describe the CCP corpus that contains data of correct and incorrect pronunciations that were essential for the training and testing processes of the proposed methods.

Abstract

Computer-Assisted Pronunciation Training (CAPT) systems aim to help a child learn the correct pronunciations of words. However, while there are many online commercial CAPT apps, there is no consensus among Speech Language Therapists (SLPs) or non-professionals about which CAPT systems, if any, work well. The prevailing assumption is that practicing with such programs is less reliable and thus does not provide the feedback necessary to allow children to improve their performance. The most common method for assessing pronunciation performance is the Goodness of Pronunciation (GOP) technique. Our paper proposes two new GOP techniques. We have found that pronunciation models that use explicit knowledge about error pronunciation patterns can lead to more accurate classification whether a phoneme was correctly pronounced or not. We evaluate the proposed pronunciation assessment methods against a baseline state of the art GOP approach, and show that the proposed techniques lead to classification performance that is more similar to that of a human expert.

Introduction

Phonological disorders are among the most prevalent communicative disabilities diagnosed in preschool and school-age children, accounting for 10% of this population (Gierut, 1998). The American Speech-Language Hearing Association determined that there is an observed relationship between early phonological disorders and subsequent reading, writing, spelling, and mathematical abilities (Castrogiovanni, 2008). Furthermore, speech production difficulties affect not only children’s communication and academic performance, but also their level of interaction with peers and adults. Considering the limited availability of speech language pathalogist (SLPs) (Squires, 2013), it is likely that a parent whose child was diagnosed with a phonological disorder would prefer to have their child practice and acquire language skills as quickly as possible, rather than relying solely on the limited time they can spend with a SLP.

Technology has provided one plausible solution to address the need of improving language skills. Advances in speech recognition technology in the early 90s (Huang et al., 1990) have attempted to address the problem of the lack of professional human resources to train children, specifically by developing computer assisted pronunciation (CAPT) systems (Witt and Young, 1997) that address children’s pronunciation disabilities. These pronunciation systems serve both as assistive tools for diagnosis as well as for practicing correct pronunciations (Witt, Young, 1997, Neri, Cucchiarini, Strik, Boves, 2002, Neri, Mich, Gerosa, Giuliani, 2008, Bunnell, Yarrington, Polikoff, 2000, Russell, Brown, Skilling, Series, Wallace, Bonham, Barker, 1996, Eskenazi, Hansma, 1998, Kawai, Hirose, September 22–25, 1997).

Unfortunately, many of the current technological tools are still limited. Despite progress in the field of speech processing, existing solutions have primarily concentrated on applying conventional automatic speech recognition (ASR) approaches to assess pronunciation in speech (Witt, Young, 1997, Neri, Cucchiarini, Strik, Boves, 2002, Ehsani, Knodt, 1998). However, conventional ASR approaches face problems in this area, as the speech of children is characterized by increased acoustic variability, while conventional ASR systems are trained to generate acoustic models from adult speech (Koenig, 2001). Moreover, some mispronunciation patterns may occur in children more frequently than in adults (Benzeghiba et al., 2007). As a result, while these automatic pronunciation systems hold promise, they are still not widely used since these technologies appear to be less reliable in terms of their performance (Ploog et al., 2013).

Other solutions have focused on explicitly modeling possible mispronunciations (Doremalen, Cucchiarini, Strik, 2013, Ronanki, Salsman, Li, 2012, Wei, Hu, Hu, Wang, 2009, Strik, Truong, De Wet, Cucchiarini, 2009); however, the state-of-the-art system using this approach has the limitation that every target pronunciation must be learned separately against its specific “competing” pronunciations (Doremalen et al., 2013) (i.e. mispronunciations). Therefore, the process of developing such a system requires substantial human supervision and input. In addition, the authors mention that in comparison to previous approaches, when introduced with low frequency pronunciation events, the system underperforms, as it requires learning of the specific features of every mispronunciation. The lack of a generalizable approach is a hurdle for automation, as is the need for a large corpus of training examples. An ideal system would incorporate knowledge of mispronunciations from a relatively small corpus while simultaneously require minimal human supervision and input.

Our research goals are derived from these limitations. The short-term objective of our research is to develop a method that will constitute the core component of an effective pronunciation analysis system for children aged 4–12 with speech sound disorders, enabling them to receive accurate feedback on their speech production, even in the absence of a clinician. The desired feedback addresses the question of whether children correctly pronounced a phoneme or not. In addition, to be effective, our system is designed to be highly automated. The long-term goal is to have such a system integrated into remediation techniques, complementing current therapy strategies. In this work, we build upon existing methodologies and extend them. Our main contributions are (1) developing an explicit model of a-priori pronunciation errors for children in the target age range, and (2) explicit modeling of the acoustics of distorted phonemes. We begin by investigating previous approaches in the field of automated pronunciation assessments. Next, we introduce a database containing mispronunciations of children with speech sound disorders. We then describe our proposed approach, and then apply a variety of different evaluation metrics to understand the strengths and weaknesses of our proposed methods. Finally, we present discussion, conclusions, and future work that is planned to further improve the current research.