BACKGROUND A new tool, "risk of bias (ROB) instrument for non-randomized studies of exposures (ROB-NRSE)," was recently developed. It is important to establish consistency in its application and interpretation across review teams. In addition, it is important to understand if specialized training and guidance will improve the reliability in the results of the assessments. Therefore, the objective of this cross-sectional study is to establish the inter-rater reliability (IRR), inter-consensus reliability (ICR), and concurrent validity of the new ROB-NRSE tool. Furthermore, as this is a relatively new tool, it is important to understand the barriers to using this tool (e.g., time to conduct assessments and reach consensus-evaluator burden). METHODS Reviewers from four participating centers will apprise the ROB of a sample of NRSE publications using ROB-NRSE tool in two stages. For IRR and ICR, two pairs of reviewers will assess the ROB for each NRSE publication. In the first stage, reviewers will assess the ROB without any formal guidance. In the second stage, reviewers will be provided customized training and guidance. At each stage, each pair of reviewers will resolve conflicts and arrive at a consensus. To calculate the IRR and ICR, we will use Gwet's AC1 statistic. For concurrent validity, reviewers will appraise a sample of NRSE publications using both the Newcastle-Ottawa Scale (NOS) and ROB-NRSE tool. We will analyze the concordance between the two tools for similar domains and for the overall judgments using Kendall's tau coefficient. To measure evaluator burden, we will assess the time taken to apply ROB-NRSE tool (without and with guidance), and the NOS. To assess the impact of customized training and guidance on the evaluator burden, we will use the generalized linear models. We will use Microsoft Excel and SAS 9.4, to manage and analyze study data, respectively. DISCUSSION The quality of evidence from systematic reviews that include NRSE depends partly on the study-level ROB assessments. The findings of this study will contribute to an improved understanding of ROB-NRSE and how best to use it.

中文翻译：

---

非接触式暴露研究偏倚风险的评估者间信度和效度：研究方案。

背景技术最近开发了一种新的工具，“用于非随机暴露研究的偏倚风险（ROB）仪器（ROB-NRSE）”。重要的是要在整个审核团队中确保其应用和解释的一致性。此外，重要的是要了解专门的培训和指导是否会提高评估结果的可靠性。因此，本横断面研究的目的是建立新的ROB-NRSE工具的评估者间可靠性（IRR），共识间可靠性（ICR）和并发有效性。此外，由于这是一个相对较新的工具，因此了解使用该工具的障碍（例如，进行评估和达到共识评估者负担的时间）非常重要。方法来自四个参与中心的审阅者将在两个阶段中使用ROB-NRSE工具来通知NRSE出版物样本的ROB。对于IRR和ICR，两对审阅者将评估每个NRSE出版物的ROB。在第一阶段，审核人员将在没有任何正式指导的情况下评估ROB。在第二阶段，将为审阅者提供定制的培训和指导。在每个阶段，每对审稿人都会解决冲突并达成共识。要计算IRR和ICR，我们将使用Gwet的AC1统计数据。为了同时有效，审阅者将使用纽卡斯尔-渥太华量表（NOS）和ROB-NRSE工具评估NRSE出版物的样本。我们将使用Kendall的tau系数分析两个工具在相似领域和整体判断上的一致性。为了评估评估者的负担，我们将评估应用ROB-NRSE工具所花费的时间（没有指导，也没有指导）。为了评估定制培训和指导对评估人员负担的影响，我们将使用广义线性模型。我们将分别使用Microsoft Excel和SAS 9.4来管理和分析研究数据。讨论包括NRSE的系统评价的证据质量部分取决于研究水平的ROB评估。这项研究的结果将有助于增进对ROB-NRSE的了解以及如何最好地使用它。讨论包括NRSE的系统评价的证据质量部分取决于研究水平的ROB评估。这项研究的结果将有助于增进对ROB-NRSE的了解以及如何最好地使用它。讨论包括NRSE的系统评价的证据质量部分取决于研究水平的ROB评估。这项研究的结果将有助于增进对ROB-NRSE的了解以及如何最好地使用它。