Skip to main content

Comparative evaluation of six nucleic acid amplification kits for SARS-CoV-2 RNA detection

Abstract

Background

SARS-CoV-2 is a newly emerged coronavirus, causing the coronavirus disease 2019 (COVID-19) outbreak in December, 2019. As drugs and vaccines of COVID-19 remain in development, accurate virus detection plays a crucial role in the current public health crisis. Quantitative real-time reverse transcriptase-polymerase chain reaction (RT-qPCR) kits have been reliably used for detection of SARS-CoV-2 RNA since the beginning of the COVID-19 outbreak, whereas isothermal nucleic acid amplification-based point-of-care automated kits have also been considered as a simpler and rapid alternative. However, as these kits have only been developed and applied clinically within a short timeframe, their clinical performance has not been adequately evaluated to date. We describe a comparative study between a newly developed cross-priming isothermal amplification (CPA) kit (Kit A) and five RT-qPCR kits (Kits B–F) to evaluate their sensitivity, specificity, predictive values and accuracy.

Methods

Fifty-two clinical samples were used including throat swabs (n = 30), nasal swabs (n = 7), nasopharyngeal swabs (n = 7) and sputum specimens (n = 8), comprising confirmed (n = 26) and negative cases (n = 26). SARS-CoV-2 detection was simultaneously performed on each sample using six nucleic acid amplification kits. The sensitivity, specificity, positive/negative predictive values (PPV/NPV) and the accuracy for each kit were assessed using clinical manifestation and molecular diagnoses as the reference standard. Reproducibility for RT-qPCR kits was evaluated in triplicate by three different operators using a SARS-CoV-2 RNA-positive sample. On the basis of the six kits’ evaluation results, CPA kit (Kit A) and two RT-qPCR Kits (Kit B and F) were applied to the SARS-CoV-2 detection in close-contacts of COVID-19 patients.

Results

For Kit A, the sensitivity, specificity, PPV/NPV and accuracy were 100%. Among the five RT-qPCR kits, Kits B, C and F had good agreement with the clinical diagnostic reports (Kappa ≥ 0.75); Kits D and E were less congruent (0.4 ≤ Kappa < 0.75). Differences between all kits were statistically significant (P < 0.001). The reproducibility of RT-qPCR kits was determined using a coefficients of variation (CV) between 0.95% and 2.57%, indicating good reproducibility.

Conclusions

This is the first comparative study to evaluate CPA and RT-qPCR kits’ specificity and sensitivity for SARS-CoV-2 detection, and could serve as a reference for clinical laboratories, thus informing testing protocols amid the rapidly progressing COVID-19 pandemic.

Introduction

With currently 59,816,510 confirmed cases and 1,410,378 deaths reported globally as of November 26th, 2020, the ongoing coronavirus disease 2019 (COVID-19) pandemic has been rapidly escalating and straining public health systems worldwide [1]. The genome sequence of the newly emerged causative agent, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2; Wuhan-Hu-1, GenBank accession number MN908947) (http://virological.org/t/novel-2019-coronavirus-genome/319) released for immediate public health support by the Global Initiative on Sharing All Influenza Data (GISAID) has shown that SARS-CoV-2 has 84% nucleotide homology with bat SARS-like coronavirus, 78% homology with the human SARS-CoV, and approximately 50% homology with Middle Eastern respiratory syndrome (MERS)-CoV [1,2,3,4]. The most commonly reported symptoms of SARS-CoV-2 infection include fever, dyspnea, leukopenia, lymphocytopenia, and ground-glass opacity commonly observed on chest computed tomography [5]. From an autopsy on the first patient who died from COVID-19, pulmonary manifestations were diffuse alveolar injury and clear membrane formation [6], which were consistent with the manifestations of acute respiratory distress syndrome and similar to the pathological features of SARS- and MERS-CoV infections [7]. In terms of public health, it has become clear that the most effective way to prevent spread of infections is by breaking the chain of transmission through social distancing.

Since the outbreak began, various commercial nucleic acid amplification kits have been developed to provide a simple and reliable means for laboratory detection of SARS-CoV-2. Among kit methodologies, real time reverse transcriptase PCR (RT-qPCR) has been the standard method for laboratory diagnosis of SARS-CoV-2 infection and has already been routinely used for detecting other respiratory viruses. According to China’s latest clinical guideline, the Diagnosis and Treatment Protocol for Novel Coronavirus Pneumonia (Trial Version 7) [8, 9], diagnosis of COVID-19 must be confirmed by RT-qPCR or gene sequencing from respiratory or blood samples, which is a key criterion for hospitalization [10, 11]. Conversely, cross-priming isothermal amplification (CPA) kits that use a closed-tube test have become a simple and fast SARS-CoV-2 test that can be performed by minimally trained personnel at point-of-care without the need for complex equipment [12]. Through specific amplification primers, fluorescent probes and DNA polymerases with high reverse transcriptase activities and chain displacement characteristics, CPA kits could complete the specific amplification process of SARS-CoV-2 fragments at a constant temperature. The fluorescence signal is detected by adaptive instruments and a real-time fluorescence curve is automatically generated.

Owing to high global demand, the development and production of nucleic acid amplification kits by various manufacturers in China have rapidly scaled up, but they have only been applied clinically for a short period of time. Hence, such kits may not have been fully validated for performance or evaluated in clinical application, where both false-negative and false-positive results have been observed [2]. Furthermore, there has been no comparative study between their analytical performance. In this study, our objective was to compare and evaluate six different commercially available nucleic acid amplification kits using clinical diagnositc reports as reference standard to provide the necessary comparative evaluation.

Materials and methods

Case definitions

In accordance with the latest guidelines of Diagnosis and Treatment of Novel Coronavirus Pneumonia (Trial Version 7) from the National Health Commission & State Administration of Traditional Chinese Medicine of China (available at http://en.nhc.gov.cn/2020-03/29/c_78469.htm), a suspected case was defined as a person with any of the following epidemiological history plus any two of the following clinical manifestations, or all three clinical manifestations if there was no clear epidemiological history. (1) Epidemiological history: within 14 days prior to the onset of illness, (a) traveled to or took residence, or (b) had contact with individuals who had fever or respiratory symptoms, in Wuhan and geographical proximities or communities where cases had been reported; (c) had contact with any laboratory confirmed cases; (d) was part of a cluster of two or more cases with fever and/or respiratory symptoms. (2) Clinical manifestations: (a) fever and/or respiratory symptoms; (b) defined imaging characteristic of SARS-CoV-2 pneumonia; (c) normal or decreased white blood cell count and/or lymphocyte count in the early stages of illness onset.

Confirmed cases were defined as suspected cases with one of the following etiological evidences: (1) a positive RT-qPCR test for SARS-CoV-2; (2) viral gene sequence highly homologous to SARS-CoV-2; (3) SARS-CoV-2 specific IgM and IgG antibodies detectable in serum, IgG detectable or reaching a titration of at least a four-fold increase during convalescence compared with the acute phase. Negative cases were defined as suspected cases with two consecutive negative nucleic acid tests taken at least 24-h apart and that SARS-CoV-2 specific IgM and IgG antibodies were negative after 7 days from illness onset.

Clinical samples

Respiratory samples (n = 52) were collected from suspected cases at the Shenzhen Third People’s Hospital and a compulsory quarantine facility in Shenzhen. Specimen types included throat swabs (n = 30), nasal swabs (n = 7), nasopharyngeal swabs (n = 7) and sputum specimens (n = 8), comprising clinical manifestationor PCR-confirmed (n = 26) and -negative (n = 26) cases.

Nucleic acid amplification

All six kits used the ORF1ab gene and the N gene as targets (ORF1ab only for Kit D) and were performed in accordance with the manufacturer’s instructions (Table 1). Briefly, in the automated CPA kit (Kit A), swab samples were directly applied to cartridges preloaded with reagents for nucleic acid purification, elution buffer and CPA reaction master mix, in which RNA were automatically purified, extracted and amplified. Fluorescence-labeled probes bind specifically to the amplified RNA targets to produce a fluorescent signal detectable by the instrument in real time for the determination of test results. For the RT-qPCR kits (Kits B–F), RNA was manually extracted with a High Pure Viral Nucleic Acid kit (Roche Diagnostics, Mannheim, Germany). For Kit B, the PCR was carried out in a 25 μL reaction volume containing 5 μL isolated DNA as template, 4 μL primer and probe mix (ORF1ab/N) and 4 μL Premix ExTaq. Amplification began with one cycle of reverse transcription at 50℃ for 10 min and a pre-denaturation step at 95 °C for 5 min, followed by 40 cycles of 95 °C for 10 s, 55 °C for 40 s. For Kit C, the PCR was carried out in a 25 μL reaction volume containing 5 μL viral cDNA as template, 19 μL RT-qPCR buffer and 1 μL enzyme mix. Amplification began with one cycle of reverse transcription at 42℃ for 15 min and a pre-denaturation step at 95 °C for 10 min, followed by 40 cycles of 95 °C for 5 s, 60 °C for 45 s. For Kit D, the 30 μL RT-qPCR reaction mixture contained 10 μL viral cDNA as template, 18.5 μL RT-qPCR buffer and 1.5 μL enzyme mix. Amplification began with one cycle of reverse transcription at 50℃ for 20 min and a pre-denaturation step at 95 °C for 10 min, followed by 40 cycles of 95 °C for 15 s, 60 °C for 30 s. For Kit E, the 20 μL RT-qPCR reaction mixture contained 4 μL viral cDNA template, 18.5 μL RT-qPCR buffer and 1.5 μL enzyme mix. Amplification began with one cycle of reverse transcription at 55℃ for 15 min and a pre-denaturation step at 95 °C for 30 min, followed by 45 cycles of 95 °C for 10 s, 60 °C for 30 s. For Kit F, the PCR was carried out in a 25 μL reaction volume containing 5 μL viral cDNA template, 17 μL RT-qPCR Buffer A and 3 μL RT-qPCR Buffer B. Amplification began with one cycle of reverse transcription at 50℃ for 15 min and a pre-denaturation step at 95 °C for 15 min, followed by 45 cycles of 94 °C for 15 s, 55 °C for 45 s. The assays were performed on an ABI 7500 Real Time PCR system (Applied Biosystems, Foster City, CA), Fluorescent signal was detected at the end of the extension step of each cycle. The results were interpreted by CT values generated using specific cut-offs for each kit (Table 1).

Table 1 Parameters for each of the six nucleic acid amplification kits evaluated

Analytical performance of six nucleic acid amplification kits

Analytical sensitivity and specificity, the positive and negative predictive values (PPV and NPV) and the accuracy for each kit were assessed using clinical diagnostic reports as the reference standard.

Reproducibility of RT-qPCR kits

Reproducibility for the RT-qPCR kits was evaluated using a positive sample verified by all kits, measured in triplicate by three different operators. The means, standard deviations (SD) and the coefficients of variation (CV) were calculated.

Statistical analysis

Statistical analyses were performed using SPSS software version 21.0 (IBM Corp., Armonk, NY). Confidence intervals (95%) were computed using the Wilson Score method. The agreement between nucleic acid amplification kits and standard diagnosis methods were compared using the McNemar-Bowker test.

Clinical application

Additional throat swabs (n = 200) from close contacts of confirmed cases were obtained from the compulsory quarantine facility to further compare Kit A with Kits B and F (approved and authorized by the National Medical Products Administration for clinical use). Agreement between three kits was assessed.

Results

Performance characteristics of six kits

From the clinical samples tested (n = 52), which comprised both confirmed (n = 26) and negative (n = 26) cases by clinical manifestation and standard RT-qPCR methods, the analytical performances from the six kits were evaluated using previous diagnosis as the reference standard (Table 2). The analytical sensitivity and NPV of all six kits were 100%. The analytical specificity and PPV of the CPA kit (Kit A) were both 100% (95% CI, 87.1–100.0), and the kappa value was 1.0 (P < 0.001). For RT-qPCR kits (Kits B–F), Kits B, C and F showed high agreement with standard diagnosis techniques (Kappa ≥ 0.75, P < 0.001), while Kits D and E were generally consistent with standard diagnosis methods (0.4 ≤ Kappa < 0.75, P < 0.001). Differences observed between the six kits were statistically significant (P < 0.001).

Table 2 Performance characteristics of six nucleic acid amplification kits for the detection of SARS-CoV-2 using clinical diagnostic reports as the reference standard

Reproducibility of RT-qPCR kits

The SDs and CVs of the fifty-two clinical samples for the five RT-qPCR kits were between 0.35 and 0.87, and 0.95 and 2.57%, respectively, indicating high reproducibility (Table 3).

Table 3 Reproducibility of five RT-qPCR kits (Kits B–F)

Clinical application

Among throat swab samples from close contacts (n = 200), one sample from a confirmed case tested positive for SARS-CoV-2 with all three (Kits A, B and F) kits, indicating total agreement between the three kits.

Discussion

COVID-19 caused by SARS-CoV-2 has now become a global pandemic and caused substantial mortality and morbidity internationally due to rapid disease propagation. The rapid and accurate identification of pathogenic virus is of great significance to the selection of appropriate treatment, saving of lives and prevention of infectious disease. To date, China has rapidly developed and approved more than 10 SARS-CoV-2 detection methods [10]. In this study, the analytical performance of six kits, including a CPA kit and five RT-qPCR kits were compared and evaluated using clinical diagnostic reports as the reference standard. RT-qPCR tests are often used in the detection pathogenic viruses in respiratory secretions and for conclusive diagnosis of COVID-19 [13]. It is a robust and reliable technique and has been widely used in laboratory tests and scientific experiments. However, RT-qPCR also has some drawbacks. For example, it could take up to 4 h or longer from initiation, including nucleic acid extraction, amplification reactions and analysis, to reporting results. Therefore, shortening the process of nucleic acid extraction and amplification time would be a useful improvement for RT-qPCR. Our evaluation of RT-qPCR kits showed that Kits B, C and F had good agreement with standard diagnosis, while Kits D and E were less consistent, all of which gave false negative results. According to an estimate, the false negative rate (FNR) of RT-qPCR may be as high as 30%–50% in real COVID-19 cases. Several factors have been proposed to explain the inconsistency or high FNR [13]. For example, this could be caused by patients who were asymptomatic or only displayed mild symptoms after infection, since the viral load of them are usually very low. The viral load sampled from patients may also vary, depending on disease progression or sample quality, which could be one of the main reasons why clinical patients can repeatedly test negative but then test positive at a later date. Therefore, nucleic acid testing should be carried out by repeated collection of specimens for suspected cases. Furthermore, primers for the target genes ORF1ab and N can be affected by virus genetic variation [13]. We have also found that sampling method, sampling depth, time of the swab stayed on the site and number of times it was rotated during sampling are related to the sample quality, which affects the results of RT-qPCR. In essence, a negative result for the first time should not be considered a confirmed laboratory diagnosis, especially in areas with high disease incidence. In contrast to RT-qPCR, CPA is a super-sensitive nucleic acid amplification method that can usually detect small amounts of RNA template with minimal sample preparation [10]. It is also often fully automated and could be used by personnel in the field or at point-of-care with minimal training, so the biosecurity risks are lower. From our evaluation, the CPA kit (Kit A) showed excellent analytical performance, with 100% sensitivity, specificity and accuracy. The main drawback of the CPA kit is a low-throughput device, which could only process two samples at once; another minor drawback in its methodology is that specific high temperatures are required.

Our current comparative evaluation also had certain limitations. Owing to the small number of confirmed cases in our study, there was a lack of sample diversity in our tests, coupled with a limited number of reagent batches used in our assays. Therefore, the limited sample size may not accurately reflect true analytical performance. During this outbreak, samples may also have been repeatedly frozen and thawed, which may have had an impact on the results. If possible, the clinical specimen sample size should be increased for more robust evaluation of the nucleic acid amplification kits for SARS-CoV-2 detection.

Conclusion

Each analysis method has its unique advantages and inevitable disadvantages. CPA has advantages of fast amplification, simple operation and convenient detection; it is an ideal candidate for the development of portable molecular diagnostics, which is suitable for field screening detection and point-of-care testing. Conversely, RT-qPCR kits are highly sensitive and specific with high-throughput capabilities, making these a more common choice for a laboratory setting. For COVID-19, we still need to develop more effective and practical methods to overcome the shortcomings of existing methods, while weighing the advantages and disadvantages of the various detection methods, to obtain the most economical and efficient choice.

Availability of data and materials

The datasets generated for this study are available on request to the corresponding author.

References

  1. Zhu N, Zhang D, Wang D, Wang W, Li X, Yang B, Song J, Zhao X, Huang B, Shi W, et al. A novel coronavirus from patients with pneumonia in China, 2019. N Engl J Med. 2020;382(8):727–33.

    Article  CAS  Google Scholar 

  2. Chu DKW, Pan Y, Cheng SMS, Hui KPY, Krishnan P, Liu Y, Ng DYM, Wan CKC, Yang P, Wang Q, et al. Molecular diagnosis of a novel coronavirus (2019-nCoV) causing an outbreak of pneumonia. Clin Chem. 2020;66(4):549–55.

    Article  Google Scholar 

  3. Lu R, Zhao X, Li J, Niu P, Yang B, Wu H, Wang W, Song H, Huang B, Zhu N, et al. Genomic characterisation and epidemiology of 2019 novel coronavirus: implications for virus origins and receptor binding. Lancet. 2020;395(10224):565–74.

    Article  CAS  Google Scholar 

  4. Yang W, Dang X, Wang Q, Xu M, Zhao Q, Zhou Y, Zhao H, Wang L, Xu Y, Wang J. Rapid Detection of SARS-CoV-2 Using Reverse transcription RT-LAMP method. medRxiv. 2020;395(10224):565.

    Google Scholar 

  5. Guan W, Ni Z, Hu Y, Liang W, Ou C, He J, Liu L, Shan H, Lei C, Hui DSC, et al. Clinical characteristics of coronavirus disease 2019 in China. N Engl J Med. 2020;28:394.

    Google Scholar 

  6. Xu Z, Shi L, Wang Y, Zhang J, Huang L, Zhang C, Liu S, Zhao P, Liu H, Zhu L, et al. Pathological findings of COVID-19 associated with acute respiratory distress syndrome. Lancet Respir Med. 2020;8(4):420–2.

    Article  CAS  Google Scholar 

  7. Chen N, Zhou M, Dong X, Qu J, Gong F, Han Y, Qiu Y, Wang J, Liu Y, Wei Y, et al. Epidemiological and clinical characteristics of 99 cases of 2019 novel coronavirus pneumonia in Wuhan, China: a descriptive study. Lancet. 2020;395(10223):507–13.

    Article  CAS  Google Scholar 

  8. Pei-Fang W. Diagnosis and Treatment Protocol for Novel Coronavirus Pneumonia (Trial Version 7). Chin Med J. 2020;133(9):1087–1095. https://doi.org/10.1097/CM9.0000000000000819.

  9. Ai T, Yang Z, Hou H, Zhan C, Chen C, Lv W, Tao Q, Sun Z, Xia L. Correlation of Chest CT and RT-PCR Testing for Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases. Radiology. 2020;296(2):E32–E40. https://doi.org/10.1148/radiol.2020200642.

  10. Shen M, Zhou Y, Ye J, Al-Maskri AAA, Kang Y, Zeng S, Cai S. Recent advances and perspectives of nucleic acid detection for coronavirus. J Pharm Anal. 2020;10(2):97–101.

    Article  Google Scholar 

  11. Zheng F, Li S, Wang S, Feng T, Jiang Z, Pan J. Cross-priming isothermal amplification combined with nucleic acid test strips for detection of meat species. Anal Biochem. 2020;597:113672.

    Article  CAS  Google Scholar 

  12. El-Tholoth M, Bau HH, Song J. A Single and Two-Stage, Closed-Tube, Molecular Test for the 2019 Novel Coronavirus (COVID-19) at Home, Clinic, and Points of Entry. ChemRxiv [Preprint]. 2020. https://doi.org/10.26434/chemrxiv.11860137.

  13. Wang Y, Kang H, Liu X, Tong Z. Combination of RT-qPCR testing and clinical features for diagnosis of COVID-19 facilitates management of SARS-CoV-2 outbreak. J Med Virol. 2020;92(6):538–9.

    Article  CAS  Google Scholar 

Download references

Acknowledgements

The authors would like to express their deepest gratitude to Dr. Zhijian Yu, Nanshan District hospital for providing eight sputum specimens. We also thank Gillian Campbell, PhD, from Liwen Bianji, Edanz Group China (www.liwenbianji.cn/ac), for editing the English text of a draft of this manuscript.

Funding

This research was supported by the National Major Science and Technology Projects of China (No. 2017ZX10303406, No. 2018ZX10714002-003–010), Sanming Project of Medicine in Shenzhen (NO.SZSM201811071).

Author information

Authors and Affiliations

Authors

Contributions

Conceived and designed the experiments: SW, QH; performed the experiments: SW, QC, MJ, YJ, LZ, LW, YL, SF, BP, WW; contributed analysis: SW, XS, RZ, HL; wrote the paper: SW, QH, PK. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Qinghua Hu.

Ethics declarations

Ethics approval and consent to participate

All samples used in this study were de-identified and anonymous to protect patient privacy and confidentiality. This study was approved by the Ethics Committee of Shenzhen Center for Disease Control and Prevention (Permit No. 2020-003A).

Consent for publication

Not applicable.

Competing interests

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, S., Shi, X., Chen, Q. et al. Comparative evaluation of six nucleic acid amplification kits for SARS-CoV-2 RNA detection. Ann Clin Microbiol Antimicrob 20, 38 (2021). https://doi.org/10.1186/s12941-021-00443-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12941-021-00443-w

Keywords