Introduction

Participation in external quality assessment (EQA) schemes to monitor the performance of diagnostic, prognostic or predictive testing in clinical routine is fundamental in a properly functioning quality system. EQA participation enables laboratories to regularly check the performance of their routine tests and to benchmark themselves against updated strategies and other laboratories. Previous international studies have indeed demonstrated that EQA schemes help laboratories to continuously improve their testing process [1,2,3,4]. In addition, laboratories can participate in EQA schemes during procedure validation and verification to prove the accuracy of the generated results.

In Belgium, medical laboratories are categorized in three types each with their own license directive: clinical biology, pathology and genetics. Molecular techniques have been widely implemented in all three laboratory types [5,6,7]. The costs for these molecular tests are only reimbursed if certain criteria are fulfilled:

  • Molecular testing performed on genetic material of micro-organisms and mentioned in article 24bis of the nomenclature listFootnote 1 of the ‘Rijksinstituut voor ziekte- en invaliditeitsverzekering/Institut National d’Assurance Maladie-Invalidité’ (RIZIV/INAMI) (e.g., genotyping of hepatitis C) should be done in a laboratory for clinical biology that is licensed by the Ministry and accredited for ISO 15189:2012 according to BELAC, the Belgian accreditation body [8,9,10].

  • Molecular testing performed on human genetic material and mentioned in article 33bis of the RIZIV/INAMI nomenclature list (e.g., detection of immunoglobulin gene or T cell receptor gene rearrangements, detection of an acquired variant in the KRAS gene) should be done in a laboratory for clinical biology or pathology that is licensed by the Ministry or in a genetics laboratory recognized by antropogenetics based on the Royal Decrete of 14.12.1987 [9, 11,12,13]. In addition, the laboratory should be accredited for ISO 15189:2012 [10].

Both according to the license criteria and ISO 15189:2012, participation to EQA is mandatory (Supplementary Table 1). For the clinical biology and pathology laboratories, Sciensano [the former ‘Wetenschappelijk Instituut Voor Volksgezondheid/Institut Scientifique de Santé Publique’ (WIV/ISP)] is responsible for the organization and follow-up of EQA. Annually, all parameters for which participation in EQA is mandatory are listed by the commissions for clinical biology and pathology. In addition, Sciensano has also founded a commission for oncology that offers a limited number of schemes in the field of solid tumors and hematological cancers. For some molecular markers, Sciensano is the responsible proficiency testing provider (e.g., for KRAS in solid tumors). For other markers, Sciensano subcontracts external EQA providers (e.g., JAK2 p.V617F). For rare or specialized assays or for parameters not provided by Sciensano, laboratories can participate in international EQA schemes, in unofficial schemes (e.g., MolecularDiagnostics.be) or they can exchange samples with other laboratories.

This current situation raises a number of questions which led to the initiation of this study, including: (1) how can you select the right EQA provider for parameters for which Sciensano does not organize an EQA? (2) is it still feasible for each laboratory to participate annually in EQA for each parameter in view of the large number of parameters to be tested and (3) if not, how can a laboratory then determine the minimum frequency? In this study, Belgian laboratories performing molecular testing in the field of microbiology, hematology and pathology were inquired for their current EQA participation strategies and their satisfaction of different EQA providers. Also, it was surveyed what the expectations of the laboratories of the EQA providers are and whether adaptations are desired. Based on these results, a guideline is proposed how to organize EQA participation in a routine setting, while still fulfilling the requirements of the Belgian law and ISO 15189:2012, and how to select a relevant EQA scheme. In addition, a proposal is made for an optimized organization of national EQA programs.

Methods

In 2017, a working group with molecular biologists affiliated to Belgian non-profit clinical laboratories, and representatives from Sciensano, BELAC and the Biomedical Quality Assurance Research Unit of the KU Leuven and UZ Leuven was founded by MolecularDiagnostics.be (a discussion forum for molecular biologists in Belgium). The working group aimed at reaching consensus guidelines for the participation in EQA programs in the field of molecular microbiology, molecular hematology and molecular pathology. There are three different types of EQA providers: those organizing EQA schemes as their core business (e.g., UK NEQAS), those organizing sporadically a limited number of schemes (e.g., EuroClonality) and those organizing sample exchanges between a limited number of participants (e.g., MolecularDiagnostics.be). The first two provider types could be ISO/IEC 17043 accredited, the last type not.

Surveys were spread to all Belgian laboratories where molecular testing is performed in the field of microbiology (N = 38), hematology (N = 29) and pathology (N = 17). The aim of this survey was to get an overview of (1) how the different laboratories in Belgium organize their EQA participation, (2) their experience with different EQA providers, (3) specific problems and (4) unmet needs of the different laboratories with regard to EQA schemes. Three separate surveys were sent for molecular microbiology, molecular hematology and molecular pathology. Each survey contained three sections: laboratory characteristics, general questions regarding the selection of an EQA provider (N = 11) and detailed questions to measure the satisfaction with specific EQA providers for a certain parameter (N = 11 per parameter and provider). During telephone calls, additional questions were asked when needed. Given the low number of participants (N = 25), data were only analyzed in a descriptive way. Based on laboratory input, the working group compared and discussed existing practices and hurdles in the performance of EQA schemes.

Results

In total, 25 molecular laboratories [microbiology (N = 13), hematology (N = 8) and pathology (N = 4)] from 14 different hospitals completed the survey regarding their EQA organization (Table 1). Participating laboratories were mainly from Flanders and affiliated to university and non-university non-profit hospitals. Different healthcare professionals are responsible for monitoring EQA results, and in most laboratories, more than one person is involved in the EQA process.

Table 1 Characteristics of the participating laboratories
Full size table

The majority of these 25 laboratories have a procedure in place for the selection of EQA providers (Table 2). Schemes using real primary patient material as well as those varying their focus throughout the years are preferred by all three laboratory groups. For molecular microbiology and hematology, schemes with a syndromic approach are preferred. All laboratories indicated that they participate in EQA schemes to verify their performance in the daily laboratory setting as much as possible. Therefore, next to scoring of analytical test results, the majority of the laboratories also ask for scoring of the clinical interpretation. In reality, the EQA schemes do not always reflect daily routine, which impacts the drive to undertake action based on EQA results (Table 3).

Table 2 Criteria for EQA provider selection and their average importance
Full size table
Table 3 Laboratory view on the use of EQA results in general
Full size table

For the most important (in number of samples or most widespread) parameters for molecular microbiology, hematology and pathology, the laboratories gave their view on commonly used EQA providers regarding sample type, number of samples, degree of difficulty, quality of the report, etc. In Table 4, an overview of available EQA schemes for molecular testing is shown. Most laboratories participate in schemes from more than one provider: for molecular microbiology the average number of providers is 2.8, molecular hematology laboratories participate in 3.4 schemes on average and molecular pathology laboratories in 3.8. Some laboratories also participate in different schemes for the same test.

Table 4 Overview of EQA schemes where survey responders participated in for molecular testing
Full size table

In total, 13 molecular microbiology laboratories gave their opinion on the following EQA providers: QCMD, Sciensano, INSTAND, CAP, UK NEQAS, SKML, NRC/UCL and MolecularDiagnostics.be. An ideal EQA scheme for molecular microbiology uses a syndromic approach (to reduce the number of EQA samples), a real patient matrix (that changes between schemes) and varies in strains (including also recent strains) and concentration. Next to this, it is important that enough material is provided to allow retesting, that the bacterial concentration is mentioned (where useful) and that the data analysis considers when different techniques are used by the participants. There was a clear preference for less samples per round and more rounds per year compared to many samples in one round.

Eight molecular hematology laboratories returned the questionnaire. They participate in EQA schemes organized by UK NEQAS, EuroClonality, SKML, Sciensano, ERIC and MolecularDiagnostics.be. Compared to microbiology and pathology, the EQA schemes sporadically organized by MolecularDiagnostics.be are commonly used. The molecular hematology laboratories prefer the presence of a clinical context to allow a proper clinical interpretation which is currently often lacking. A clear, simple laboratory-specific report showing the laboratory’s performance status over time is appreciated. There is a need to expand quantitative schemes as well as the use of more challenging samples.

Although only four molecular pathology laboratories completed the survey, they agree on positive and negative points of EQA schemes they participated in (ESP, Sciensano, EMQN, cIQc, CAP, UK NEQAS, NordiQC and GenQA). Schemes providing in time extensive personalized reports including a comparison of the participating laboratories and schemes offering a plan of action after a bad result are preferred. Non-representative samples (e.g., cell lines), too little variation in target and absence of challenging samples are common problems of several EQA schemes in the field of molecular pathology.

Since Sciensano is the responsible party for national EQA follow-up for parameters covered by articles 24, 24bis and 33bis of the RIZIV nomenclature list, they annually provide a list which was agreed by the commissions of clinical biology and pathology with parameters for which participation in EQA programs is mandatory. According to the laboratories in molecular hematology, microbiology and pathology, most but not all relevant markers are present (Table 5). Therefore, 40 % of the laboratories would like an extension of the list. Since some laboratories feel that the current situation is not ideal to test their daily performance, the following proposal was phrased: ‘Let’s assume that laboratories would have a choice of a limited number of approved EQA providers to choose for each reimbursed parameter, and Sciensano would still be responsible for the registration, payment of the registration fee and national follow-up. Would you prefer this situation over the current situation?’ Of all questioned laboratories, 12 laboratories answered ‘yes,’ 7 ‘no’ and 1 claimed that national follow-up is irrelevant as they are the only laboratory testing for certain rare parameters. The laboratories which answered ‘no’ all gave this answer because Sciensano already selected the provider of their choice. Elements deemed most important for approving EQA providers include: clinically relevant sample types and matrices and variation in target concentration (Table 5).

Table 5 Feedback on the proposal for a new national model
Full size table

Discussion

In terms of laboratory accreditation, most requirements and guidelines listed in Supplementary Table 1 require participation in EQA schemes [8, 10, 14,15,16,17,18,]. These sources do not specify how to select an EQA provider nor do they define the frequency of participation. This study surveyed Belgian laboratories to identify their current local EQA strategy and their needs, to come to an overview what molecular laboratories look for in an EQA scheme and to formulate a proposal how to manage their EQA participation.

One major limitation of the study is that only one-third of the invited laboratories completed the survey. Possible explanations could be the extensiveness of the questionnaire, high workload of people working in molecular testing laboratories or the perception that giving their opinion is useless since participation in EQA is often considered as something mandatory. That is why we also want to raise awareness about the importance of selecting a EQA provider that fulfills the laboratories’ needs with this paper. Although there are less laboratories doing molecular testing in Wallonia (24 %) and Brussels (12 %) as compared to Flanders (63 %), there still is some overrepresentation of Flemish laboratories in our study. Nevertheless, the answers reflect the problematic points of the participation in EQA and some general trends were observed. In general, laboratories have a high willingness to participate in EQA schemes, not only because it is mandatory, but also because they want to know their performance level and improve it if necessary. The received surveys indicate that there is room for improvement in the selection of EQA schemes to better fit the laboratories’ needs. On the other hand, laboratories might not participate in the EQA program that is best suited. Although guidelines exist for organizing EQA programs (e.g., ISO/IEC 17043), there are still large differences between the accredited EQA providers. The majority of laboratories already have a selection procedure for EQA schemes in place. Participation in a particular scheme can be mandatory by law (for clinical biology and pathology) or is guided by the distribution of real patient material, the use of a syndromic approach and challenging samples.

Proposed workflow for EQA organization in a molecular laboratory

Based on the survey and an already existing guideline for clinical chemistry [18], we identified the following critical elements to consider when selecting an EQA provider:

1. Accreditation

As required by multiple sources, it is recommended to choose a provider that is ISO/IEC 17043 accredited

Note 1: If a non-accredited provider is chosen, laboratories should have a good reason and document this

Note 2: If EQA schemes (accredited or non-accredited) are not available, other forms of interlaboratory comparison (e.g., sample exchange between laboratories) are recommended

2. Sample type

Select an EQA provider that distributes samples which are fit for purpose

 If daily performance is to be evaluated, providers that send real patient material are preferred

 If the limitations of the detection method are to be evaluated, artificial samples in a matrix resembling the routine matrix could be used (e.g., paraffin-embedded cell lines)

 If reader accuracy or interpretation of the test outcome is to be evaluated, digital cases are also an option

Note: It is also advised to evaluate pre-analytical and post-analytical processes as much as possible. At the moment, a limited number of EQA schemes assessing the pre-analytical phase is available, although pre-analytical errors impact the whole upstream testing process

3. Target range

EQA providers distributing samples containing a clinically relevant range of concentrations or allelic frequencies are preferred. If only strongly positive cases are included, the laboratory will not have a clear insight on its performance

Note: National guidelines regarding clinical relevance of tested targets should be considered when participating in an international EQA scheme

4. Target variation

Select a provider that includes a variance in clinically relevant targets, either within one parameter or within a disease group (‘syndromic approach’)

Example 1: EQA schemes for RAS testing in molecular pathology that only include mutations in codons 12 or 13 of the RAS gene should be avoided

Example 2: EQA schemes for influenza virus with recently/currently circulating strains are preferred

Example 3: EQA schemes for myeloproliferative neoplasms where JAK2 p. V617F and CALR are tested in parallel can be preferred

5. Frequency

The frequency of the selected EQA scheme should best match the ‘ideal participation frequency’ of the laboratory that is identified based on a risk analysis, as described below [15]

6. Educational value

EQA schemes should be sufficiently educational. This implies that challenging cases could be included but also that the report should be clear and give sufficient explanation on the test outcomes. In addition, EQA schemes providing feedback on the interpretation of test results are preferred

7. Number of participants

If the laboratory wants to benchmark itself, it is advisable to select an EQA provider with a high number of participants. A better overview of performance related to the techniques used might then be given in the final report

To be able to follow these steps, it is essential that the commissions for clinical biology and pathology of Sciensano as well as the laboratories know which are the available EQA schemes. To help them make an informed choice, EQA providers must be transparent about the material they distribute and the service they provide.

Since there are no sources specifying the minimal required frequency of participation in EQA schemes, BELAC and Eurachem [14, 19] stated that the frequency should be based on a risk analysis considering the following elements;

1. Number of samples annually tested

A previous study has indicated that analyzing a higher number of samples is correlated with more experience and less errors [1]

2. Possibility to group tests based on disease or on technique

Example 1: Make sure that a panel for next-generation sequencing (NGS) is included once a year in an EQA program (e.g., year 1 for colon, year 2 for lung, etc.)

Example 2: Participate annually in an EQA scheme for respiratory pathogens

Note: As a laboratory you have to trust that the EQA provider will alter the variants annually, so all pathogens will be tested over a larger period of time

3. The number of involved operators and frequency of turnover in technical staff

It is good practice that each operator participates in at least one EQA scheme each year

4. The education level and general experience of the personnel

For starting operators, participation in EQA schemes could be part of their training program

5. The availability of certified reference materials

If certified reference materials are available and continuously used as internal controls, the need for frequent EQA participation might be lower

6. The complexity of the measurement technique

High complex multi-step testing methods [e.g., next-generation sequencing (NGS)] might be more error-prone then single-step tests and thus require more stringent quality control measures

7. The level of criticality of the result and its final use

Tests of which results alter the duration of a patient’s life or quality of life should be handled with more care

A re-evaluation of the defined participation frequency has to be performed in case of important changes in laboratory organization. Moreover, 3 years is considered as the maximum time interval between two EQA participations.

In Belgium, most laboratories still annually participate in EQA for each offered test. For larger testing centers and laboratories using big NGS panels, this becomes untenable. Grouping parameters per sub-discipline can help. A sub-discipline is then defined as ‘an area of technical competence defined by a minimum of one Measurement Technique, Property and Product, which are related,’ meaning tests can be performed with the same training of the personnel [15]. Furthermore, tests that are used for several years with no change in setup or interpretation may not need an annual EQA. The following EQA participation strategy is used by one of the study participants: for new tests the laboratory participates annually. If during the first 3 years no major genotyping or interpretation errors (i.e., errors which could harm patients) are made, participation is from then on only required once every 3 years. It is also considered that every laboratory technique (cfr. ‘sub-discipline’ as per above) is each year covered by an EQA, and that each technical responsible person participates once per year.

Use of EQA results

One important purpose of EQA is to educate laboratories. For laboratories, it is important, to the extent possible, to treat EQA samples the same way as patient samples. EQA providers also have a responsibility, by drafting a clear final report and by providing sufficient individual feedback. It is important that the laboratory has a procedure for communicating EQA results within the laboratory (e.g., with laboratory management or during the management review) and for taking actions. With a score varying between 2.9 and 3.0 on 4 in the three disciplines, respectively, laboratories feel quite encouraged by EQA providers to take actions. Surprisingly, in only 31 % of the surveyed laboratories in molecular microbiology and 13 % in molecular hematology, the quality manager is (co)-responsible for the follow-up of the results, whereas this is 100 % in molecular pathology laboratories.

Proposal for a national model for organizing external quality assessment

The results of the surveys done in this study showed that improvements to the current situation of mandatory EQA participation are desired.

Now, laboratories are participating in schemes that do not always coincide with their defined needs in terms of frequency of participation, sample type, target range, etc. To benefit from the nationally organized schemes, for each parameter at least 2 different schemes should be offered with a comparable degree of difficulty, if available. As such, every laboratory can decide which scheme best fits their needs (e.g., syndromic approach versus single parameter test). In addition, more parameters should be included in the list for which EQAs are nationally offered to avoid that laboratories opt for an easy EQA for the missing parameters. It is very important that the educational value of these schemes should prevail the fact that participation is mandatory. The commissions for clinical biology and pathology and the commission for oncology have with these results the information to discuss the possibilities to answer to these desires and to communicate the wishes to the EQA organizers. This publication can thus be a tool for the commissions to select the most appropriate EQA provider, if this is not available to encourage the providers to develop more appropriate EQAs.

There was a clear overlap between the three disciplines (molecular microbiology, hematology and pathology) with regard to ‘important elements for selecting an EQA scheme.’ Representative samples, syndromic approach, variation in target and concentration/allelic frequency, scoring of clinical interpretation and a simple personalized report delivered in time with a detailed comparison of different laboratories and techniques are highly requested. For each discipline, there were some extra recurrent elements such as decreasing the number of samples per round in some microbiology EQA schemes, and the need for more quantitative schemes in the hematology setting.

To limit the number of annual EQA participations for tests with an obligatory accreditation, laboratories should perform a risk analysis to determine for each parameter the frequency of EQA participation. This approach, supported by BELAC, is contradictory with the annual compulsory EQA participation by the Belgian law for tests requiring accreditation. It is therefore requested to mitigate this obligation and give the responsibility to the laboratories to define the frequency of EQA participation.

Conclusion

This study aimed to create an overview of how Belgian laboratories currently handle their participation in EQA programs, to survey whether they are satisfied and to propose a new national model for the selection of EQA providers and follow-up of laboratory performance by the different national commissions. Although participation in EQA programs is mandatory according to national and international guidelines, no clear requirements exist for the frequency of participation and the selection criteria for EQA providers. Since participation in EQA is in Belgium regulated by a national body, laboratories are currently not always participating to the most relevant scheme for their daily routine. Therefore, the authors propose a new model, offering laboratories more options. To conclude, in this paper guidance is offered to determine the frequency for participation in EQA schemes and to select the most relevant EQA provider.