Inter-observer agreement (IOA) is a key aspect of data quality in time-and-motion studies of clinical work. To date, such studies have used simple and ad hoc approaches for IOA assessment, often with minimal reporting of methodological details. The main methodological issues are how to align time-stamped task intervals that rarely have agreeing start and end times, and how to assess IOA for multiple nominal variables. We present a combination of methods that simultaneously addresses both these issues and provides a more appropriate measure by which to assess IOA for time-and-motion studies. The issue of alignment is addressed by converting task-level data into small time windows then aligning data from different observers by time. A method applicable to multivariate nominal data, the iota score, is then applied to the time-aligned data. We illustrate our approach by comparing iota scores to the mean of univariate Cohen’s kappa scores through application of these measures to existing data from an observational study of emergency department physicians. While the two scores generated very similar results under certain conditions, iota was more resilient to sparse data issues. Our results suggest that iota applied to time windows considerably improves on previous methods used for IOA assessment in time-and-motion studies, and that Cohen’s kappa and other univariate measures should not be considered the gold standard. Rather, there is an urgent need for ongoing explicit discussion of methodological issues and solutions to improve the ways in which data quality is assessed in time-and-motion studies in order to ensure the conclusions drawn from such studies are robust.

观察者之间的协议（IOA）是临床工作的时空研究中数据质量的关键方面。迄今为止，此类研究已使用简单且临时的方法进行IOA评估，通常很少报告方法学细节。主要的方法论问题是如何调整很少有一致开始和结束时间的时间戳任务间隔，以及如何评估多个名义变量的IOA。我们提出了可以同时解决这两个问题的方法的组合，并提供了一种更合适的方法来评估时间和运动研究的IOA。通过将任务级别的数据转换为较小的时间窗口，然后按时间对齐来自不同观察者的数据，可以解决对齐问题。然后将适用于多元标称数据的方法（iota得分）应用于时间对齐的数据。通过将这些指标应用于急诊科医师观察性研究的现有数据，我们通过将iota得分与单变量Cohen's kappa得分的平均值进行比较来说明我们的方法。尽管在一定条件下这两个分数产生了非常相似的结果，但是iota在稀疏数据问题上更具弹性。我们的结果表明，适用于时间窗口的iota大大改进了以前的时间和运动研究中用于IOA评估的方法，并且Cohen的kappa和其他单变量测量方法不应被视为黄金标准。相反，迫切需要对方法论问题和解决方案进行持续的明确讨论，以改善在时间与运动研究中评估数据质量的方式，以确保从此类研究得出的结论是可靠的。