A slowdown or a speedup in response times across experimental conditions can be taken as evidence of online deployment of knowledge. However, response-time difference measures are rarely evaluated on their reliability, and there is no standard practice to estimate it. In this article, we used three open data sets to explore an approach to reliability that is based on mixed-effects modeling and to examine model criticism as an outlier treatment strategy. The results suggest that the model-based approach can be superior but show no clear advantage of model criticism. We followed up these results with a simulation study to identify the specific conditions in which the model-based approach has the most benefits. Researchers who cannot include a large number of items and have a moderate level of noise in their data may find this approach particularly useful. We concluded by calling for more awareness and research on the psychometric properties of measures in the field.

实验条件下响应时间的减慢或加快可以作为知识在线部署的证据。然而，响应时间差异度量很少评估其可靠性，并且没有标准实践来估计它。在本文中，我们使用三个开放数据集来探索一种基于混合效应建模的可靠性方法，并检查模型批评作为异常值处理策略。结果表明，基于模型的方法可能更优越，但模型批评没有显示出明显的优势。我们通过模拟研究跟踪这些结果，以确定基于模型的方法具有最大优势的具体条件。无法包含大量项目且数据中噪音水平适中的研究人员可能会发现这种方法特别有用。最后，我们呼吁人们更多地认识和研究该领域措施的心理测量特性。