Think-aloud protocols are a highly valued usability testing method for identifying usability problems. Despite the value of conducting think-aloud usability test sessions, analyzing think-aloud sessions is often time-consuming and labor-intensive. Consequently, previous research has urged the community to develop techniques to support fast-paced analysis. In this work, we took the first step to design and evaluate machine learning (ML) models to automatically detect usability problem encounters based on users’ verbalization and speech features in think-aloud sessions. Inspired by recent research that shows subtle patterns in users’ verbalizations and speech features tend to occur when they encounter problems, we examined whether these patterns can be utilized to improve the automatic detection of usability problems. We first conducted and recorded think-aloud sessions and then examined the effect of different input features, ML models, test products, and users on usability problem encounters detection. Our work uncovers several technical and user interface design challenges and sets a baseline for automating usability problem detection and integrating such automation into UX practitioners’ workflow.

中文翻译：

---

自动检测在大声思考会话中遇到的可用性问题

大声思考协议是一种用于识别可用性问题的高价值可用性测试方法。尽管进行有声思考可用性测试会话很有价值，但分析有声思考会话通常是耗时且劳动密集型的。因此，以前的研究敦促社区开发技术来支持快节奏的分析。在这项工作中，我们迈出了设计和评估机器学习 (ML) 模型的第一步，以根据用户在大声思考会话中的语言表达和语音特征自动检测遇到的可用性问题。受最近研究显示用户语言表达和语音特征在遇到问题时往往会出现微妙模式的启发，我们检查了这些模式是否可用于改进可用性问题的自动检测。我们首先进行并记录了大声思考会话，然后检查了不同输入特征、ML 模型、测试产品和用户对可用性问题遇到检测的影响。我们的工作揭示了几个技术和用户界面设计挑战，并为自动化可用性问题检测和将此类自动化集成到 UX 从业者的工作流程中设定了基线。