Usability testing is an essential element in human–computer interaction studies, and its core purpose is not only to evaluate usability in general but also to identify specific problems. Synchronous usability testing methods are costly to conduct on a large scale, as well as subject to time constraints. Thus, in realistic practice, asynchronous testing is often performed and usually adopts an automatic logging approach to identify usability problems based on collected data. However, in prior research, an in-depth human evaluation review phase has been required to examine each user interaction for usability problems. This consumes considerable time, and the results are likely to be subjective. In this study, we propose a novel method to identify and prioritize usability problems quantitatively using an explainable neural network approach and conduct an experiment based on device usage logs collected in a usability test. Using an explainable neural network, we assess user interactions for their relative influence on an overall usability score, predicting each interaction’s probability of causing a usability problem. Assigning usability scores to each task, we model the relative importance of each individual interaction by calculating a weight for each based on feature maps output by the neural network. Then, we identify usability problems by reviewing the interactions showing the highest importance weight values. The experimental results indicate that our proposed method effectively identifies usability problems and demonstrate that it performs quantitatively better compared to other benchmark methods.

中文翻译：

---

异步测试环境下基于可解释神经网络的可用性问题识别

可用性测试是人机交互研究中的一个基本要素，其核心目的不仅是评估总体可用性，而且是识别具体问题。同步可用性测试方法大规模进行的成本很高，并且受时间限制。因此，在实际实践中，通常会执行异步测试，并且通常采用自动记录方法来根据收集的数据识别可用性问题。然而，在先前的研究中，需要一个深入的人工评估审查阶段来检查每个用户交互是否存在可用性问题。这会耗费大量时间，而且结果很可能是主观的。在这项研究中，我们提出了一种新方法，使用可解释的神经网络方法定量识别可用性问题并确定其优先级，并根据可用性测试中收集的设备使用日志进行实验。使用可解释的神经网络，我们评估用户交互对整体可用性分数的相对影响，预测每个交互导致可用性问题的概率。为每个任务分配可用性分数，我们通过基于神经网络输出的特征图计算每个交互的权重来模拟每个交互的相对重要性。然后，我们通过查看显示最高重要性权重值的交互来识别可用性问题。