Inter-laboratory testing as a strategy for external quality assessment for qualitative detection of SARS-CoV-2 by real-time RT-PCR testing in India

India has a network of 124 viral research and diagnostic laboratories (VRDLs), under the umbrella of Department of Health Research and Indian Council of Medical Research (DHR-ICMR) that has the capability to diagnose 20-25 viral aetiologies. When COVID-19 struck India in January 2020, the VRDLs paved the way for scaling up the diagnostic capabilities across India1. The VRDL network provided a platform for escalating laboratory capacity from 115 laboratories in March 2020 to 2349 laboratories as on January 22, 20212. Among these laboratories, 1331 were testing SARS-CoV-2 by an open system real-time reverse transcription-PCR (rRT-PCR) machine with the use of any rRT-PCR kit. This test methodology, also the gold standard of testing3, has been the mainstay of COVID-19 diagnosis in India. This is a complex test whose accuracy depends on well-defined laboratory practices and quality system. Given the massive number of laboratories doing COVID-19 testing using a myriad of rRT-PCR kits, reliability and accuracy of results have always been a concern.

To ensure the reliability of these tests, quality control (QC) of testing was of paramount importance. Initially, the ICMR-NIV Pune played a role of the QC laboratory1, but with increasing number of testing laboratories, it was difficult for ICMR-NIV, Pune to conduct QC activity for all the testing laboratories. Therefore, QC laboratories within each State/regions were identified. The strategy for QC of COVID-19 testing across these laboratories was to deploy external quality assessment (EQA) by comparing results of testing laboratories with results reported by the designated QC laboratories of samples submitted by the testing laboratories. EQA is considered as an effective tool for reliability of results reported by clinical diagnostic laboratories4. The results of this inter-laboratory quality control (ILQC) activity are presented here. Challenges and experience with implementing inter-laboratory comparison across such a wide network of COVID-19 testing laboratories are also discussed.

Operational strategy

This study was conducted from September 2020 till November 2020 and involved 1017 laboratories. These laboratories were empanelled up to October 15, 2020 with ICMR5 for COVID-19 testing by open system rRT-PCR test. The data were entered into national portal of ICMR during October and November 2020. The participation was voluntary, and thus, 597 testing laboratories participated during the study period. The activity was designed as a three-tiered structure – testing laboratories, regional QC laboratories and national QC laboratory.

Of all the COVID-19 testing laboratories, 34 laboratories were chosen and designated as regional QC laboratories. The regional QC laboratories were scattered across the country so as to cover all the testing laboratories. These laboratories participated in monthly QC with NIV, Pune, wherein coded aliquots of five negative and five positive samples were sent. The results were analyzed manually and the laboratories were chosen based on their 100 per cent concordance with ICMR-NIV in ILQC activity for SARS-CoV-2 undertaken from March till August. All the testing laboratories were mapped to regional QC laboratories that rechecked the samples sent by COVID-19 testing laboratories All the regional QC laboratories sent their samples to national QC laboratory at ICMR-NIV that is a WHO-designated reference laboratory and participates in global QC for COVID-196. Thus, ICMR-NIV served as both regional and national QC laboratory.

Samples: Considering the stability of RNA virus, each laboratory selected and sent five positive samples with cycle threshold (Ct) values between 25 and 30 and five negative samples of COVID-19, taken within the last 15 days, to the laboratory above them in the tier. Samples with low Ct values were discouraged as their results were mostly unequivocal. Each sample was oral or nasopharyngeal swab. Sample volume was between 500 µl and 1.0 ml and was stored at 4°C till transported to all QC laboratories. The laboratories coded the samples and entered the relevant details into an online portal before shipping. Samples were accepted by regional QC laboratories and national QC laboratory if these were in a good condition and were rejected if the samples were found leaked or volume was insufficient or there was a mismatch between the sample code displayed on the portal and on aliquot physically received. These laboratories were asked to send another lot of samples to QC laboratory.

Testing: All the samples by each laboratory in the tier were treated as typical patient sample. For testing, each laboratory used the in-use RNA extraction and rRT-PCR kit/platform. A total of 63 different RNA extraction kits and 50 different rRT-PCR kits were used by the laboratories. The different rRT-PCR kits could be categorized into three categories: (i) Commercial U.S. Food and Drug Administration (USFDA) approved7; (ii) Commercial non-USFDA approved but validated by ICMR and approved by Drug Controller General of India for use in India. All these kits had a sensitivity of 95 per cent or more and specificity of 99 per cent or more8; and (iii) Non-commercial ICMR-NIV rRT-PCR kit. This is a laboratory-developed test based on a combination of Berlin and Hong Kong protocol1. The primers and probes including enzyme and master mix were procured from Thermo Fisher Scientific, Invitrogen Bioservices India Pvt Ltd.

All private laboratories used commercial kits, while non-commercial ICMR-NIV rRT-PCR kit was mainly supplied to public laboratories.

Software development: An open-source web-based application (COVID QA/QC software), a Linux-based software package/tool was developed for the management and analysis of QA-QC data collected through multiple laboratories across India. The System was hosted on Centos7 operating system using Apache webserver. The tool was developed in a modular architecture with the prevailing version offering the different features to the users:

(i) Standardized data collection based on a standardized list of parameters.

(ii) Fully configurable system through the super-admin (ICMR) module.

(iii) Role-based access – Each stakeholder could access page as per their role (section Data acquisition and analysis).

(iv) Security and audit trail – All the accounts were password protected, and patient parameters were encrypted in the database. Strong firewall protection was also enabled.

(v) Key data points were entered to the portal – RNA extraction kit used, RNA extraction platform, rRT-PCR kit used, rRT-PCR platform and sample details.

(vi) Ct values for each gene were captured for each sample, but results were interpreted on a qualitative basis (positive, negative). Based on the qualitative analysis, the web portal calculated per cent concordance for each laboratory and participating laboratories could visualize results.

Data acquisition and analysis: The COVID QA/QC software captured, stored and analyzed the results reported by each participating laboratory. The testing laboratories filled in the sample details and results in the portal. Similar details were entered by the QC laboratories. Both the results were visible to ICMR, which were marked as concordant or dis-concordant. The report was visible online to both the QC and testing laboratories, thus enabling transparency in the entire process.

Findings & conclusions

Of the 597 testing laboratories that participated during the study period, on a qualitative basis, majority (513 or 86%) had a concordance of 90 per cent and above while remaining 84 laboratories (14%) scored 80 per cent or below concordance. It was not possible to point out the absolute reasons for discordance seen in the results of 14 per cent of participating laboratories. It could have been because all steps of testing differed between laboratories and sensitivity of each step could have affected the results.

When analyzed by nature of laboratories, almost identical number of public and private laboratories had similar concordance levels. Of the 597 testing laboratories, 286 were public and 311 were private laboratories which were NABL accredited. Of these 597 laboratories, 424 passed with 100 per cent concordance [207 (48.8%) public laboratories and 217 (51.2%) private laboratories]. Eighty nine laboratories scored 90 per cent concordance. Of these 89 samples (1 sample from each of the 89 laboratories), 50 samples were reported as false positives and 39 as false negative by the testing laboratories. Remaining laboratories in both the sectors had 80 per cent or less concordance. These 84 laboratories submitted 230 samples that were discordant. Given the similar performance of laboratories in both the public and private sectors, it appeared that factors responsible for discordance were common and related to variability of testing materials used by the laboratories rather than the competency of the laboratories to run the test.

The results were also interpreted in terms of rRT-PCR kits and types of RNA extraction platform used by the laboratories. Since different laboratories used different rRT-PCR kits, a matrix was designed and the results were analyzed for nine combinations (Table). It was seen that irrespective of the kit combination used, around 85-90 per cent laboratories in each combination met the performance criteria of concordance of 90 per cent and above, except for combination number 4 in the Table. Laboratories in combination numbers 6 and 9 were more of an aberration, and this could possibly be because only a few laboratories used this kit combination. To address this, it was decided that all QC laboratories would use ICMR-NIV kit in the next cycle. This will help decipher performance characteristics of different testing kits, and if any test system has an advantage over other system given the fact that sensitivity of each system varies.

Manual extraction of RNA was done by majority (524) of testing laboratories (450 – 86% met the scoring criteria and 74 failed) while 73 (63 – 86% met the scoring criteria and 10 laboratories failed) used automated system for RNA extraction. Similarly, majority of QC laboratories did manual extraction of RNA while rest did automated extraction. It was clear from these data that the sensitivity of testing kits rather than the platform used or nature (type) of laboratory played a critical role in determining the quality of SARS-CoV-2 rRT-PCR results.

Forthcoming plans

This ILQC activity showcased quality of diagnosis of SARS-CoV-2 in India. This exercise assured the precision and accuracy of the results reported by the laboratories that tested SARS-CoV-2 samples in India by rRT-PCR method. The EQA for SARS-CoV-2 testing, using different methods, has also been implemented in various countries to understand the reliability of rRT-PCR testing910. Our results indicated that SARS-CoV-2 testing in India was done optimally (based on the concordance criteria) and was in-line with what was being done globally. The WHO has provided proficiency testing (PT) panels for all COVID-19 rRT-PCR laboratories in India. This has been distributed to most laboratories that participated in this study. Results of the evaluation of this PT panel will further augment the results reported in this study.