Many studies of speech perception assess the intelligibility of spoken sentence stimuli by means of transcription tasks (‘type out what you hear’). The intelligibility of a given stimulus is then often expressed in terms of percentage of words correctly reported from the target sentence. Yet scoring the participants’ raw responses for words correctly identified from the target sentence is a time-consuming task, and hence resource-intensive. Moreover, there is no consensus among speech scientists about what specific protocol to use for the human scoring, limiting the reliability of human scores. The present paper evaluates various forms of fuzzy string matching between participants’ responses and target sentences, as automated metrics of listener transcript accuracy. We demonstrate that one particular metric, the token sort ratio, is a consistent, highly efficient, and accurate metric for automated assessment of listener transcripts, as evidenced by high correlations with human-generated scores (best correlation: r = 0.940) and a strong relationship to acoustic markers of speech intelligibility. Thus, fuzzy string matching provides a practical tool for assessment of listener transcript accuracy in large-scale speech intelligibility studies. See https://tokensortratio.netlify.app for an online implementation.

许多语音感知研究通过转录任务（“输入你听到的内容”）来评估口语刺激的可理解性。然后，给定刺激的可理解性通常以从目标句子中正确报告的单词的百分比来表示。然而，对参与者对从目标句子中正确识别的单词的原始响应进行评分是一项耗时的任务，因此需要大量资源。此外，语音科学家对于人类评分使用什么特定协议没有达成共识，这限制了人类评分的可靠性。本论文评估参与者的反应和目标句子之间的各种形式的模糊字符串匹配，作为听众转录准确性的自动指标。我们证明了一个特定的指标，即令牌排序率，是一致的，r =  0.940) 并且与语音清晰度的声学标记密切相关。因此，模糊字符串匹配为评估大规模语音清晰度研究中的听者转录准确性提供了实用工具。有关在线实施，请参阅 https://tokensortratio.netlify.app。