Objectives To investigate the validity of data extraction in systematic reviews of adverse events, the effect of data extraction errors on the results, and to develop a classification framework for data extraction errors to support further methodological research. Design Reproducibility study. Data sources PubMed was searched for eligible systematic reviews published between 1 January 2015 and 1 January 2020. Metadata from the randomised controlled trials were extracted from the systematic reviews by four authors. The original data sources (eg, full text and ClinicalTrials.gov) were then referred to by the same authors to reproduce the data used in these meta-analyses. Eligibility criteria for selecting studies Systematic reviews were included when based on randomised controlled trials for healthcare interventions that reported safety as the exclusive outcome, with at least one pair meta-analysis that included five or more randomised controlled trials and with a 2×2 table of data for event counts and sample sizes in intervention and control arms available for each trial in the meta-analysis. Main outcome measures The primary outcome was data extraction errors summarised at three levels: study level, meta-analysis level, and systematic review level. The potential effect of such errors on the results was further investigated. Results 201 systematic reviews and 829 pairwise meta-analyses involving 10 386 randomised controlled trials were included. Data extraction could not be reproduced in 1762 (17.0%) of 10 386 trials. In 554 (66.8%) of 829 meta-analyses, at least one randomised controlled trial had data extraction errors; 171 (85.1%) of 201 systematic reviews had at least one meta-analysis with data extraction errors. The most common types of data extraction errors were numerical errors (49.2%, 867/1762) and ambiguous errors (29.9%, 526/1762), mainly caused by ambiguous definitions of the outcomes. These categories were followed by three others: zero assumption errors, misidentification, and mismatching errors. The impact of these errors were analysed on 288 meta-analyses. Data extraction errors led to 10 (3.5%) of 288 meta-analyses changing the direction of the effect and 19 (6.6%) of 288 meta-analyses changing the significance of the P value. Meta-analyses that had two or more different types of errors were more susceptible to these changes than those with only one type of error (for moderate changes, 11 (28.2%) of 39 v 26 (10.4%) 249, P=0.002; for large changes, 5 (12.8%) of 39 v 8 (3.2%) of 249, P=0.01). Conclusion Systematic reviews of adverse events potentially have serious issues in terms of the reproducibility of the data extraction, and these errors can mislead the conclusions. Implementation guidelines are urgently required to help authors of future systematic reviews improve the validity of data extraction. A subset of the data can be found at /. The dataset could be obtained from the first author (xuchang2016@runbox.com) or the corresponding author (svohra@ualberta.ca) on request.

中文翻译：

---

不良事件证据合成实践中数据提取的有效性：再现性研究

目的 研究不良事件系统评价中数据提取的有效性、数据提取错误对结果的影响，并开发数据提取错误的分类框架以支持进一步的方法学研究。设计再现性研究。数据来源 PubMed 搜索了 2015 年 1 月 1 日至 2020 年 1 月 1 日之间发表的符合条件的系统评价。随机对照试验的元数据是从四位作者的系统评价中提取的。然后，同一作者引用原始数据源（例如全文和 ClinicalTrials.gov）来重现这些荟萃分析中使用的数据。选择研究的资格标准 当基于医疗保健干预措施的随机对照试验（将安全性作为唯一结果）时，包括系统评价，至少有一对荟萃分析，其中包括五项或更多随机对照试验，并有一个 2×2 表荟萃分析中每项试验的干预组和对照组的事件计数和样本量数据。主要结果指标 主要结果是数据提取错误，概括为三个级别：研究级别、荟萃分析级别和系统评价级别。进一步研究了此类错误对结果的潜在影响。结果 纳入 201 项系统评价和 829 项配对荟萃分析，涉及 10 386 项随机对照试验。10 386 项试验中的 1762 项（17.0%）无法重现数据提取。在 829 项荟萃分析中的 554 项（66.8%）中，至少一项随机对照试验存在数据提取错误；201 项系统评价中有 171 项 (85.1%) 至少有一项荟萃分析存在数据提取错误。最常见的数据提取错误类型是数值错误（49.2%，867/1762）和模糊错误（29.9%，526/1762），主要是由于结果定义不明确造成的。这些类别之后是其他三个类别：零假设错误、错误识别和不匹配错误。通过 288 项荟萃分析分析了这些错误的影响。数据提取错误导致 288 项荟萃分析中的 10 项 (3.5%) 改变了效应方向，以及 288 项荟萃分析中的 19 项 (6.6%) 改变了 P 值的显着性。具有两种或多种不同类型错误的荟萃分析比仅具有一种类型错误的荟萃分析更容易受到这些变化的影响（对于中等变化，39 中的 11 例（28.2%） vs 26 例（10.4%）249 例，P=0.002；对于较大的变化，39 例中有 5 例（12.8%） vs 249 例中有 8 例（3.2%），P=0.01）。结论 不良事件的系统评价在数据提取的可重复性方面可能存在严重问题，这些错误可能会误导结论。迫切需要实施指南来帮助未来系统评价的作者提高数据提取的有效性。数据的子集可以在以下位置找到/。该数据集可根据要求从第一作者 (xuchang2016@runbox.com) 或通讯作者 (svohra@ualberta.ca) 获取。