Composite reference standards are used to evaluate the accuracy of a new test in the absence of a perfect reference test. A composite reference standard defines a fixed, transparent rule to classify subjects into disease positive and disease negative groups based on existing imperfect tests. The accuracy of the composite reference standard itself has received limited attention. We show that increasing the number of tests used to define a composite reference standard can worsen its accuracy, leading to underestimation or overestimation of the new test’s accuracy. Further, estimates based on composite reference standards vary with disease prevalence, indicating that they may not be comparable across studies. These problems can be attributed to the fact that composite reference standards make a simplistic classification and then ignore the uncertainty in this classification. Latent class models that adjust for the accuracy of the different imperfect tests and the dependence between them should be pursued to make better use of data
Composite reference standards define a fixed, transparent rule to classify subjects into disease positive and disease negative groups based on existing imperfect tests
They are widely regarded as appropriate for determining sensitivity and specificity of a new test in the absence of a perfect reference test
Though a composite reference standard is attractive for its simple and transparent construction, it can result in biased estimates as it makes suboptimal use of data
Bias due to a composite reference standard can worsen as more information is gathered and the new test’s accuracy can be overestimated if the errors made by the composite reference standard and the new test are correlated
Composite reference standards cannot aid standardisation across settings when disease prevalence varies
Appropriately constructed latent class models should be used to make complete use of the information gathered from multiple imperfect tests
For many diseases, a perfect diagnostic test may not exist or cannot be applied owing to costs or ethical concerns. Researchers evaluating a new test for the disease have no perfect reference against which to compare it. For example, GeneXpert (Xpert) is a nucleic acid amplification test for paediatric pulmonary tuberculosis (TB). Culture is an inadequate reference standard owing to its poor sensitivity,1 so the accuracy of Xpert is often evaluated by comparing it to a composite reference standard based on multiple imperfect tests.23
Table 1 shows a composite reference standard defined using data gathered in a South African cohort study of symptomatic children.6 It classifies a child who is positive on culture, smear microscopy, chest radiography, or the tuberculin skin test as having TB. The apparent advantage of this composite reference standard is that it would identify more TB cases than culture alone. Studies using a standard such as this typically treat it as an error-free reference test to estimate the new test’s sensitivity (proportion of all patients with the disease that are correctly detected by the new test) and specificity (proportion of all patients without the disease who are correctly detected by the new test).37 Ensuring the new test is not used to define the composite reference standard is thought to protect against overestimating the new test’s accuracy.8
A composite reference standard for childhood TB compared with the results of a latent class analysis in a cohort of 749 children
Composite reference standards are used for diverse conditions including Chlamydia trachomatis infection, apathy, and prostate cancer (see web table 1). Some are required by regulatory authorities for approval of new tests.9 Despite their widespread use, the number of tests necessary to define an adequate standard is unknown. Composite reference standards based on two or three tests are most common,710 but some are based on upwards of eight or nine tests (web table 1).211
Consequences of both the composite reference standard and the new test making the same errors are also not well understood. We discuss previously ignored concerns related to composite reference standards using numerical examples and data from a study of childhood TB and draw attention to better methods. We focus on composite reference standards based on the OR rule, which classifies patients with at least one positive test as disease positive and those with all negative tests as disease negative.3 Other possible composite rules include the AND rule, which classifies a patient as disease positive only if all tests are positive, or K positive rules, which classify a patient as disease positive only if at least K tests are positive.12
In these examples, the sensitivities of the component tests used to define the composite reference standard are moderate to high and their specificities are near perfect. An OR rule composite reference standard is, therefore, anticipated to have higher sensitivity than a single imperfect reference test.
We generated data for a sample of 1000 people assuming the composite reference standard was made up of component tests with sensitivity of 70% and specificities of 98-100% (detailed explanation in the web appendix). Disease prevalence was assumed to be 10%—that is, 100 patients were disease positive and 900 were disease negative. The new test under evaluation was set to have a sensitivity of 90% and a specificity of 98%.
Depending on their accuracy, adding more component tests to the composite reference standard might cause more misclassification rather than less. We start with the ideal situation where each component test has perfect specificity of 100% and where the different component tests are conditionally independent (meaning that they are not prone to making the same false positive or false negative errors). Table 2 lists the frequency of results on the component tests and the composite reference standards based on them. As we move from a single reference test to a composite reference standard based on two or three component tests, the number of patients correctly classified as having the disease increases from 70 (34+15+15+6) to 91 (34+15+15+6+15+6) to 97 (34+15+15+6+15+6+6). The gain at the second step is less than at the first. After about five component tests (data not shown), additional tests increase costs but don’t result in any gain, as all 1000 patients are correctly identified.
Expected frequency of results on individual tests and OR rule based composite reference standards in 1000 subjects*
Table 2 shows how misclassification changes when the specificity of the component tests decreases to 98%. The classification of disease positive patients remains the same, but the number of misclassified disease negative patients increases from 17 to 51 as we move from a single reference test to a composite reference standard with three component tests. For this example, the composite reference standard with three component tests resulted in more misclassified patients overall (three false positives and 51 false negatives) than the single reference test (30 false positives and 17 false negatives). The overall number of misclassified patients will continue to rise as more component tests are added to the composite reference standard.12
Using the data in table 2 we can show that when all component tests have perfect specificity and are conditionally independent, the sensitivity of the new test is estimated accurately at its value of 90% irrespective of the number of tests in the composite reference standard (fig 1a). The estimate of the specificity of the new test steadily improves with every added test until it reaches the true value (fig 1b).
Change in estimates of sensitivity and specificity of the new test with increasing number of component tests in the composite reference standard.
When the specificity of the component tests falls to 98%, the specificity estimates of the new test are almost identical to those obtained previously, but the sensitivity of the new test is now underestimated (fig 1). When the composite reference standard was composed of three tests, for example, the estimated sensitivity was 59%, much lower than the true value of 90%. This underestimation worsens with every test added to the composite reference standard.
So far, we have assumed that all tests—component tests in the composite reference standard and the new test—are conditionally independent. In practice, however, errors made by multiple tests might be correlated.13 In studies evaluating new tests for Chlamydia trachomatis, for example, the component tests and the new test are typically nucleic acid amplification tests, which risk making the same false positive error of detecting a non-viable organism.14 In these situations, the composite reference standard can overestimate the accuracy of the new test because it does not adjust for the presence of conditional dependence—that is, the errors of the new test remain undetected.
To study the effects of conditional dependence, we generated data from a setting where both tests are likely to make the same false positive errors even though their specificity remains high at 98% (see web table 2 for details).6 As in our previous example, the sensitivity of the new test is underestimated, and this worsens with each component test added to the composite reference standard (fig 1). The specificity of the new test is underestimated when compared to a single imperfect test but becomes overestimated as component tests are added to the composite reference standard (fig 1). As the number of component tests increases, the estimated specificity of the new test converges to a value higher than the true value.12 In our example, the new test’s specificity will converge at 99.94%, compared with its true specificity of 98%. This may not seem like a large magnitude of bias but could lead to an important underestimate of the number of false positives the new test will produce in a low prevalence population.
When a new test is compared to the same composite reference standard in two different studies, the reported value of the new test’s accuracy will depend on the disease prevalence in each study. Using a standardised composite reference standard therefore does not ensure comparability across studies. We considered a composite reference standard based on three conditionally independent component tests, each with sensitivity 70% and specificity 98%. Disease prevalence ranged from low (5%) to high (30%), as might be expected across geographic regions or healthcare settings. The new test’s sensitivity was assumed to be 90% and its specificity 98%, as before. We found the estimated sensitivity of the new test ranged from 43% when the prevalence was 5% to 79% when the prevalence is 30% (web figure 1). Estimated specificity did not vary greatly with prevalence for the settings we used.
The drawbacks of the composite reference standard can be overcome using a statistical modelling approach called latent class analysis.3 Instead of classifying subjects into fixed disease categories, latent class analysis estimates the probability that each patient has the disease using all observed tests, including the test under evaluation (web figure 2). It adjusts for the sensitivity and specificity of each test as well as the possibility of conditional dependence between them. Simply put, latent class analysis considers how certain we are about classifying patients into diseased or non-diseased groups rather than making a black and white decision. Column 8 of table 1 shows the estimated risk of TB for each observed combination of tests based on a recent latent class analysis for the childhood TB data.6
Notably, the estimated risk of TB from this latent class analysis follows the gradation proposed by an expert group’s clinical case definition (column 9 of table 1),4 which classifies subjects into confirmed TB, probable TB, possible TB, and unlikely TB groups, using the same four tests as the composite reference standard. Our data show that composite reference standard would classify confirmed, probable and possible TB cases all as having TB, resulting in an estimated prevalence of 94%. Using culture as a reference would only consider the confirmed TB cases, resulting in an underestimate of the prevalence (16.4%). Latent class analysis estimates a 100% risk of TB for most cases of confirmed TB, though the risk is lower for unusual patterns. The risk of TB among the probable and possible TB cases ranges from 9% to 52% among patients with a negative Xpert test but increases when Xpert is positive. The resulting prevalence estimate based on latent class analysis is 26.7%. Because the latent class analysis adjusts for conditional dependence between culture and Xpert, it also provides a more realistic estimate of Xpert sensitivity (49.4%) than would be obtained with culture (74.4%) or the composite reference standard (22.5%) (see web material for how the latent class analysis estimates were calculated).
The advantages of latent class analysis are accompanied by the challenges of using a more sophisticated analytical technique. Construction of these models requires interdisciplinary expertise of both methodologists and clinicians6 to determine the particular tests, covariates, conditional dependence structure, and previous knowledge to be considered. Validation of these models against a perfect reference may not always be possible. Sometimes competing models cannot be distinguished using standard statistical methods.15 This is not a drawback of latent class analysis, but a reflection of the uncertainty in our knowledge due to the lack of a perfect reference test. Comparison with external information, such as the experts’ clinical case definition, can aid in assessing whether the model provides sensible results. This step is important because, as with all statistical models, incorrect model specification can lead to biased results.16
Composite reference standards are considered valid for estimating diagnostic accuracy when no perfect reference standard exists.37 But we have shown that the OR rule based composite reference standard leads to biased estimates of the accuracy of a new test unless stringent conditions hold. The additional information gathered from each component test results in worsening bias. Our previous work has shown these observations also apply to composite reference standards based on the AND rule and or the K positive rule.12
Composite reference standards may be considered clinically meaningful17 as they resemble clinical decision rules, which classify patients into mutually exclusive categories to support decision making—for example, rules identifying whether a subject is a candidate for TB treatment. Such decision rules are not necessary in research settings as no black or white decision needs to be made. Clinical decision rules might indicate the best possible management strategy, but are recognised by clinicians as imperfect.16 Yet similar rules are used to define composite reference standards for a diagnostic accuracy studies with no such recognition.
In the absence of a perfect reference test, a new test could be evaluated in terms of outcomes such as diagnostic yield or effect on patient management instead of accuracy.18 Latent class analysis would also be relevant in such analyses to estimate percentage of overdiagnosis or overtreatment,619 eventually supporting the development of optimal clinical decision rules.
As our ability to measure results on multiple tests/biomarkers increases, development of appropriate latent class models should be pursued in the absence of a perfect reference test to make optimal use of the data gathered.
More detail on the problems with composite reference standards https://www.ncbi.nlm.nih.gov/pubmed/26555849
A review paper on use of latent class models for diagnostic research https://www.ncbi.nlm.nih.gov/pubmed/24272278
Applications of latent class models with free accompanying software https://www.ncbi.nlm.nih.gov/pubmed/27737841, https://www.ncbi.nlm.nih.gov/pubmed/7840100
Guidelines for reporting latent class models https://www.equator-network.org/reporting-guidelines/stard-blcm/
Contributors and sources: The authors include biostatisticians, epidemiologists, and clinicians with expertise in diagnostic research and a particular interest in methods for evaluating diagnostic accuracy in the absence of a perfect reference test. All authors participated in planning and writing the paper. IS generated the numerical examples. ND is the guarantor.
Funding: This work was supported by funding from the Canadian Institutes of Health Research (Grant number 89857).
Competing interests: All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/coi_disclosure.pdf and declare support from the Canadian Institutes of Health Research for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years, no other relationships or activities that could appear to have influenced the submitted work.

中文翻译：

---

对诊断研究中的复合参考标准的担忧

在没有完善的参考测试的情况下，使用复合参考标准评估新测试的准确性。复合参考标准定义了一个固定的透明规则，可以基于现有的不完善测试将受试者分为疾病阳性和疾病阴性组。复合参考标准本身的准确性受到了有限的关注。我们表明，增加用于定义复合参考标准的测试数量可能会降低其准确性，从而导致对新测试准确性的低估或高估。此外，基于综合参考标准的估计值会随疾病患病率而变化，这表明它们在各个研究之间可能不可比。这些问题可归因于以下事实：复合参考标准进行了简单的分类，然后忽略了该分类中的不确定性。应当针对潜在缺陷模型进行调整，以适应不同缺陷测试的准确性以及它们之间的依赖性，从而更好地利用数据
复合参考标准定义的固定的，透明的规则进行分类受试者入疾病阳性和基于现有不完善测试疾病负性基团
它们被广泛认为是适当的，用于确定在不存在一个完美的基准测试的一个新的测试的灵敏度和特异性
虽然复合参考标准以其简单透明的结构而吸引人，它可能导致偏差的估计，因为它对数据的使用不够理想。
由于收集了更多的信息，复合参考标准导致的偏差可能会恶化，如果产生错误，新测试的准确性可能会被高估。通过综合参考标准与新测试相关联
当疾病患病率发生变化时，复合参考标准不能在所有环境中实现标准化，
应使用适当构建的潜在分类模型来充分利用从多个不完善测试中收集的信息。
对于许多疾病，由于成本原因，可能不存在完美的诊断测试或无法使用完美的诊断测试或道德上的顾虑。评估针对该疾病的新检测方法的研究人员没有完美的参考来与之进行比较。例如，GeneXpert（Xpert）是针对小儿肺结核（TB）的核酸扩增测试。由于灵敏度低，文化是不足的参考标准，1因此，经常通过将Xpert与基于多种不完善测试的复合参考标准进行比较来评估Xpert的准确性。23
表1显示了使用在有症状儿童的南非队列研究中收集到的数据定义的综合参考标准。6它对文化，涂片显微镜检查，胸部X线摄影或结核菌素皮肤试验阳性的儿童分类为TB。该综合参考标准的明显优势在于，与单独培养相比，它将识别出更多的结核病病例。使用此类标准的研究通常将其作为无差错的参考测试，以评估新测试的敏感性（通过新测试正确检测出的所有患有该疾病的患者的比例）和特异性（所有未患有该疾病的患者的比例） 37通过确保不使用新测试来定义复合参考标准，可以防止过高估计新测试的准确性。8
与749名儿童的潜在分类分析结果相比较的儿童结核病综合参考标准。该
复合参考标准用于多种疾病，包括沙眼衣原体感染，情感淡漠和前列腺癌（请参见网络表1）。监管机构要求其中一些批准新测试。9尽管已广泛使用，但定义适当标准所必需的测试数量尚不得而知。基于两个或三个测试的综合参考标准是最常见的710，但有些基于八个或九个以上的测试（Web表1）。211
复合参考标准和产生相同误差的新测试的后果也未得到很好的理解。我们使用数字示例和对儿童结核病研究的数据讨论了以前与复合参考标准有关的关注点，并提请注意更好的方法。我们将重点放在基于OR规则的综合参考标准上，该标准将至少一项阳性测试的患者分类为疾病阳性，将所有阴性测试的患者分类为疾病阴性。3其他可能的综合规则包括AND规则，该规则将患者分类为疾病仅在所有检测均呈阳性或K呈阳性的规则时才呈阳性，仅当至少K呈阳性时才将患者分类为疾病阳性12。
在这些示例中，用于定义复合参考标准的成分测试的敏感性中等至很高，并且其特异性接近完美。因此，预期OR规则复合参考标准比单个不完善的参考测试具有更高的灵敏度。
我们假设1000人的样本产生了数据，假设复合参考标准品由成分测试组成，敏感性为70％，特异性为98-100％（网络附录中有详细说明）。假定疾病患病率为10％，即100例患者为阳性，900例为阴性。评估中的新测试的灵敏度设定为90％，特异性为98％。
根据其准确性，在复合参考标准中添加更多的组件测试可能会导致更多的错误分类，而不是更少的错误分类。我们从理想的情况开始，每个组件测试具有100％的完美特异性，并且不同的组件测试在条件上是独立的（这意味着它们不易产生相同的假阳性或假阴性错误）。表2列出了成分测试结果的频率以及基于这些结果的综合参考标准。当我们从基于两种或三种成分测试的单一参考测试转变为复合参考标准时，正确分类为患有该疾病的患者人数从70（34 + 15 + 15 + 6）增加到91（34 + 15 + 15 + 6 + 15 + 6）至97（34 + 15 + 15 + 6 + 15 + 6 + 6）。第二步的增益小于第一步。
单个测试和基于OR规则的复合参考标准的预期结果在1000个主题中的出现*
表2显示了当成分测试的特异性降低到98％时，分类错误如何变化。疾病阳性患者的分类保持不变，但随着我们从单一参考测试变为具有三项成分测试的复合参考标准，被错误分类为疾病阴性的患者的数量从17增至51。在此示例中，与三项成分测试相比，综合参考标准导致的患者总体分类错误（三项假阳性和51项假阴性）比单项参比测试（30项假阳性和17项假阴性）更多。随着更多的成分测试被添加到复合参考标准中，分类错误的患者总数将继续增加12。
使用表2中的数据，我们可以证明，当所有成分测试均具有完美的特异性且在条件上独立时，无论复合参考标准中的测试数量如何，新测试的灵敏度均准确地估计为90％的值（图1a）。新测试特异性的估计随着每个附加测试的稳步提高，直到达到真实值为止（图1b）。
随着复合参考标准中组分测试数量的增加，新测试的敏感性和特异性估计值也随之变化。
当成分测试的特异性下降到98％时，新测试的特异性估计值几乎与以前获得的估计值相同，但是新测试的灵敏度现在被低估了（图1）。例如，当复合参考标准物由三个测试组成时，估计的灵敏度为59％，远低于90％的真实值。将每个测试添加到复合参考标准后，这种低估都会加剧。
到目前为止，我们已经假定所有测试（复合参考标准中的组件测试和新测试）在条件上都是独立的。然而，实际上，可能会将多个测试所导致的错误联系起来。13在评估沙眼衣原体新测试的研究中例如，成分检测和新检测通常是核酸扩增检测，可能会导致检测非活生物体时出现相同的假阳性错误。14在这种情况下，复合参考标准可能会高估新检测的准确性。测试，因为它不会针对条件依赖性的存在进行调整-也就是说，新测试的错误仍未被发现。
为了研究条件依赖性的影响，我们从一个环境中生成了数据，即使它们的特异性仍保持在98％的高水平，这两种测试仍可能产生相同的假阳性错误（有关详细信息，请参见网络表2）。6与前面的示例一样，新测试的灵敏度被低估了，并且每添加到复合参考标准品中的每个组件测试都会使这种情况恶化（图1）。与单个不完善的测试相比，新测试的特异性被低估了，但是随着组分测试被添加到复合参考标准中，其特异性被高估了（图1）。随着组分测试数量的增加，新测试的特异性估计值收敛到高于真实值的值。12在我们的示例中，新测​​试的特异性收敛到99.94％，而其真实特异性为98％。
当在两项不同的研究中将一项新测试与相同的复合参考标准进行比较时，新测试准确性的报告值将取决于每项研究中的疾病患病率。因此，使用标准化的复合参考标准不能确保各个研究之间的可比性。我们考虑了基于三个有条件独立成分测试的复合参考标准，每个测试的敏感性为70％，特异性为98％。疾病发生率的范围从低（5％）到高（30％），正如跨地理区域或医疗机构所期望的那样。像以前一样，新检测的敏感性被假定为90％，特异性为98％。我们发现，新检测的估计灵敏度范围从流行率为5％时的43％到流行率为30％时的79％（网络图1）。
可以使用称为潜在类别分析的统计建模方法来克服复合参考标准的弊端。3潜在类别分析不是将受试者分类为固定的疾病类别，而是使用所有观察到的测试（包括测试）来估计每个患者患病的可能性。评估中（网络图2）。它针对每个测试的敏感性和特异性以及它们之间条件依赖的可能性进行调整。简而言之，潜在类别分析考虑了我们对将患者分类为疾病或非疾病组而不是做出黑白决定的确定性。表1的第8栏显示了根据最近对儿童结核病数据的潜在分类分析得出的每种观察到的测试组合的估计结核病风险。6
值得注意的是，根据这种潜在类别分析得出的估计结核病风险遵循专家组临床病例定义（表1第9列）4提出的等级，将患者分为已确诊的结核病，可能的结核病，可能的结核病和不太可能的结核病组，使用与复合参考标准相同的四个测试。我们的数据表明，复合参考标准将已确诊，可能和可能的结核病病例归类为患有结核病，估计患病率为94％。用文化作为参考仅会考虑确诊的结核病例，从而低估了患病率（16.4％）。潜在类别分析估计，在大多数确诊结核病例中，结核风险为100％，尽管在异常情况下，该风险较低。Xpert测试阴性的患者中，可能和可能的TB病例中TB的风险范围为9％至52％，但当Xpert呈阳性时，结核病的风险会增加。根据潜在类别分析得出的患病率估算为26.7％。由于潜在类别分析针对文化和Xpert之间的条件依赖性进行了调整，因此与文化（74.4％）或复合参考标准（22.5％）相比，它也提供了对Xpert敏感性的更实际估计（49.4％）（请参阅网络有关如何计算潜在类别分析估算值的材料）。
潜在类分析的优点伴随着使用更复杂的分析技术的挑战。这些模型的构建需要方法学家和临床医生6的跨学科专业知识，以确定特定的测试，协变量，条件依赖性结构和要考虑的先前知识。并非总是可以针对完美的参考来验证这些模型。有时，无法使用标准的统计方法来区分竞争模型。15这并不是潜在类别分析的缺点，而是由于缺乏完善的参考测试而导致我们知识的不确定性。与外部信息（例如专家的临床病例定义）进行比较，可以帮助评估模型是否提供合理的结果。这一步很重要，因为，
当不存在完善的参考标准时，复合参考标准被认为对估计诊断准确性有效。37但是，我们已经证明，除非满足严格的条件，否则基于OR规则的复合参考标准会导致对新测试准确性的估计有偏差。从每个组件测试中收集的其他信息会导致偏差恶化。我们以前的工作表明，这些观察结果也适用于基于AND规则和/或K正则规则的复合参考标准。12
复合参考标准类似于临床决策规则，可以被认为具有临床意义17，该规则将患者分为相互排斥的类别以支持决策制定，例如，确定受试者是否为结核病治疗候选者的规则。这样的决策规则在研究环境中不是必需的，因为不需要做出黑白决策。临床决策规则可能指示最佳的管理策略，但被临床医生认为是不完善的。16然而，类似的规则用于定义诊断准确性研究的复合参考标准，而没有这种认可。
在没有完善的参考测试的情况下，可以根据结果的准确性（例如诊断率或对患者管理的影响）来评估新测试，而不是准确性。18潜在类别分析也可能与此类分析相关，以估计过度诊断或过度治疗的百分比，619最终支持了最佳临床决策规则的开发。
随着我们在多种测试/生物标记物上测量结果的能力增强，应在没有完美的参考测试的情况下开发合适的潜在分类模型，以最佳地利用收集到的数据。
有关复合参考标准的问题的更多详细信息https://www.ncbi.nlm.nih.gov/pubmed/26555849
关于使用潜在类模型进行诊断研究的评论文章https://www.ncbi.nlm.nih.gov/pubmed/24272278
潜在类模型与免费随附软件https：//www.ncbi.nlm.nih的应用。 gov / pubmed / 27737841，https://www.ncbi.nlm.nih.gov/pubmed/7840100
报告潜在类模型的指南https://www.equator-network.org/reporting-guidelines/stard-blcm/
贡献者资料来源：作者包括在诊断研究方面具有专业知识的生物统计学家，流行病学家和临床医生，并且对缺乏完善参考测试的情况下评估诊断准确性的方法特别感兴趣。所有作者都参与了计划和撰写论文。IS生成了数值示例。ND是担保人。
资金：这项工作得到了加拿大卫生研究院（拨款号89857）的资助。
竞争利益：所有作者均已完成ICMJE统一披露表，网址为http://www.icmje.org/coi_disclosure.pdf，并声明加拿大卫生研究院对提交的工作表示支持；在过去三年中，没有与可能对提交的工作有兴趣的任何组织建立财务关系，也没有其他似乎对提交的工作有影响的关系或活动。