BACKGROUND The Cochrane Bias Methods Group recently developed the "Risk of Bias (ROB) in Non-randomized Studies of Interventions" (ROBINS-I) tool to assess ROB for non-randomized studies of interventions (NRSI). It is important to establish consistency in its application and interpretation across review teams. In addition, it is important to understand if specialized training and guidance will improve the reliability of the results of the assessments. Therefore, the objective of this cross-sectional study is to establish the inter-rater reliability (IRR), inter-consensus reliability (ICR), and concurrent validity of ROBINS-I. Furthermore, as this is a relatively new tool, it is important to understand the barriers to using this tool (e.g., time to conduct assessments and reach consensus-evaluator burden). METHODS Reviewers from four participating centers will appraise the ROB of a sample of NRSI publications using the ROBINS-I tool in two stages. For IRR and ICR, two pairs of reviewers will assess the ROB for each NRSI publication. In the first stage, reviewers will assess the ROB without any formal guidance. In the second stage, reviewers will be provided customized training and guidance. At each stage, each pair of reviewers will resolve conflicts and arrive at a consensus. To calculate the IRR and ICR, we will use Gwet's AC1 statistic. For concurrent validity, reviewers will appraise a sample of NRSI publications using both the New-castle Ottawa Scale (NOS) and ROBINS-I. We will analyze the concordance between the two tools for similar domains and for the overall judgments using Kendall's tau coefficient. To measure the evaluator burden, we will assess the time taken to apply the ROBINS-I (without and with guidance), and the NOS. To assess the impact of customized training and guidance on the evaluator burden, we will use the generalized linear models. We will use Microsoft Excel and SAS 9.4 to manage and analyze study data, respectively. DISCUSSION The quality of evidence from systematic reviews that include NRS depends partly on the study-level ROB assessments. The findings of this study will contribute to an improved understanding of the ROBINS-I tool and how best to use it.

中文翻译：

---

ROBINS-I的评分者间信度和并发有效性：横断面研究规程。

背景技术Cochrane Bias方法小组最近开发了“非随机干预研究中的偏倚风险（ROB）”（ROBINS-I）工具，用于评估ROB用于非随机干预研究（NRSI）。重要的是要在整个审核团队中确保其应用和解释的一致性。此外，重要的是要了解专门的培训和指导是否会提高评估结果的可靠性。因此，本横断面研究的目的是建立ROBINS-I的评定者间可靠性（IRR），共识间可靠性（ICR）和并发有效性。此外，由于这是一个相对较新的工具，因此了解使用该工具的障碍（例如，进行评估和达到共识评估者负担的时间）非常重要。方法来自四个参与中心的审阅者将在两个阶段中使用ROBINS-I工具评估NRSI出版物样本的ROB。对于IRR和ICR，两对审阅者将评估每个NRSI出版物的ROB。在第一阶段，审核人员将在没有任何正式指导的情况下评估ROB。在第二阶段，将为审阅者提供定制的培训和指导。在每个阶段，每对审稿人都会解决冲突并达成共识。要计算IRR和ICR，我们将使用Gwet的AC1统计数据。为了同时有效，审阅者将使用新堡渥太华量表（NOS）和ROBINS-I评估NRSI出版物的样本。我们将使用Kendall的tau系数分析两个工具在相似领域和整体判断上的一致性。为了评估评估人员的负担，我们将评估应用ROBINS-I（无指导和无指导）和NOS所花费的时间。为了评估定制培训和指导对评估人员负担的影响，我们将使用广义线性模型。我们将分别使用Microsoft Excel和SAS 9.4来管理和分析研究数据。讨论包括NRS的系统评价的证据质量部分取决于研究水平的ROB评估。这项研究的结果将有助于增进对ROBINS-I工具及其最佳使用方式的理解。4分别管理和分析研究数据。讨论包括NRS的系统评价的证据质量部分取决于研究水平的ROB评估。这项研究的结果将有助于增进对ROBINS-I工具及其最佳使用方式的理解。4分别管理和分析研究数据。讨论包括NRS的系统评价的证据质量部分取决于研究水平的ROB评估。这项研究的结果将有助于增进对ROBINS-I工具及其最佳使用方式的理解。