Computerized registry as a potential tool for surveillance and management of complex bone and joint infections in France

Description of the CRIOAc network

To better manage C-BJI, a national network of referral centres for C-BJIs ("Centre de reference des infections ostéo-articulaires complexes" (CRIOAc)) was set up in 2008 by the Ministry of Health.⁵ Since January 2009, these centres have been implemented sequentially throughout the French territory and their missions have been specified. In 2011, nine CRIOAcs and 15 associate centres were certified.⁶ Their roles are: to provide advice on the management of BJI at a local and regional level, especially for C-BJI, through periodic multidisciplinary meetings (MM) involving at least an infectious diseases specialist, an orthopaedic surgeon, and a microbiologist; and to organize learning sessions for BJI management and to promote clinical research in the field.

Several points were included in the initial call for tender: 1) decision support in MM and monitoring of patient outcomes regardless of their site of care; 2) coordination of care and sharing of patient files among the centres; 3) monitoring of the centres' activity; and 4) production of epidemiological data. For this purpose, a national secure online information system was implemented in early 2012 in order to aggregate the data from the different MM to each referral centre. Recorded data are syntheses of each individual MM, based on the criteria used to define C-BJI including surgical and medical advices.

The national database is hosted by an external accredited society (Inovelan, Lille, France). The data warehouse ensures activity analyses and duplicates management. The system was endorsed by the French National Data Protection Agency in 2012 ("Commission nationale Informatique et Libertés" (CNIL)/2012-220). Since 2016, a national scientific council has been in charge of defining priorities, controlling data recorded in the registry, and promoting research, based on post-processing and database quality assessment to provide informative data.⁷ Funding of this registry and the referral centres is supported by the French Ministry of Health.

Database design

For each patient, an individual electronic case report form (eCRF) based on the common data grid is filled after each MM by a dedicated agent in each centre. If needed, additional data are manually extracted from the medical records for the purpose of the MM.

Variables recorded in the eCRF are detailed in Supplementary Table i. They include: characteristics of the MM; demographics; clinical and biological data; characteristics of the infection; and decision of MM (recommended surgery, proposed antibiotic treatment, free fields to specify the surgical or medical recommendation, definition as a C-BJI or not, and criterion of definition for complexity).

Population registry

The national registry is automatically filled with these eCRFs. Each new patient receives a unique identification number for duplicate management. For one patient, if multiple sites are involved in the same infection episode presented in MM, one line by site of infection is filled in the MM database. Each centre has secured access to visualize its own data, and records are frozen after validation. Metrics are automatically extractable but without any quality assessment of the reported data.

The database was cleaned and reorganized for analysis purpose. Each component encoded as a list of variables in the native eCRF was separated into independent variables (categorical variables of medical history). As a consequence, if an item was not specifically checked in the list, the data were automatically considered missing.

History of cognitive impairment and excessive alcohol consumption were extracted using a basic natural language processing system of unstructured data, as they were not in the initial eCRF. Obesity was defined as a body mass index (BMI) above 30 kg/m², kidney impairment as a calculated glomerular filtration rate under 30 ml/minute/1.73 m²; these were corrected from calculation on numerical data when available. Outliers were deleted.

Quality assessment of the registry

To assess the quality of the national registry, a gold standard dataset was created for comparison, by checking the electronic complete medical charts based on a random sample of ten to 20 patients examined in a MM from each referral centre (first patient presented during various predefined MM in each centre between 2014 and 2016).

The variables chosen to assess the quality are summarized in Supplementary Table i. Specific definitions were given as necessary: creatinine level in μmol/l; recent CRP level (< one month) in mg/l; diabetes mellitus defined by the necessity of specific therapy; obesity defined as a BMI over 30 kg/m²; tobacco use defined as active smoking or stopped for less than three years; excessive alcohol consumption defined as more than 30 g/day; immunodeficiency as the use of immunosuppressive drug, chemotherapy, or biotherapy, presence of AIDS, or solid organ or stem cell transplantation; neoplasia defined as solid cancer or haemopathy, excluding remission > five years; heart failure defined as clinical acute heart failure, or an ejection fraction < 30% measured by echography; kidney failure as a glomerular filtration rate lower than 30 ml/mn/1.73 m² or necessity of dialysis; and liver failure as presence of cirrhosis or acute liver failure. Excessive alcohol consumption and cognitive impairment were added in the controlled dataset in order to test the reliability of the natural language processing extraction method.

Data completeness and consistency were evaluated by comparing the data registry to the random gold standard dataset. Data completeness was defined by mean exhaustivity of collected data, shown as mean exhaustivity of each variable (%) and number of missing data. Internal completeness was evaluated for each variable as the percentage of available data in the national registry compared to the dataset. Data consistency was measured by checking the likeness of contents between medical chart and registry eCRF. For each variable, consistency was evaluated as the percentage of identical variables in both databases when data were available. There is no consensual threshold in the literature and we proposed a threshold for acceptable global mean consistency at 80%, as proposed elsewhere for epidemiological registries.^7-9 Mean data consistency and completeness were calculated for each centre.

A complementary sensitivity data analysis completeness and consistency was performed, considering missing data for categorical variables of medical history in the national database as negative results. Indeed, we only considered these missing data, artificially missing due to the structure of the eCRF.

A descriptive data analysis on the whole registry was performed using R software version 3.5.1 (R Foundation for Statistical Computing, Vienna, Austria).

Description of the database

Between October 2012 and December 2016, 38,076 lines were recorded in the database, corresponding to 30,607 presentations in MM and to 17,748 individual patients. Overall 17,941 patient cases were presented in MM at the nine CRIOAcs, and 12,667 in the 15 associate centres (Figure 1). The mean number of patients presented in each MM was 7.8 (SD 4.73). An increase was observed between 2012 and 2016, becoming steady at around eight patients between 2015 and 2016 (8.3 and 8.7, respectively).

Fig. 1

Map of referral centres for bone and joint infections in France. Total number of patients presented in multidisciplinary meetings (MM) between 2012 and 2016 in the nine CRIOAcs ("Centre de Référence pour les Infections Osteo-Articulaires Complexes") and the 15 associate centres.

Few patients were recorded in the database in 2012, but the total number of patients increased throughout the years, until stabilization at about 8,500 presentations per year in 2015 and 2016. There were important variations between centres in terms of activity, according to their type (CRIOAc or associate centre) and number of MM organized each year (median 42 per year (interquartile range 17.2 to 49.0)).

Multiple presentations in MM for individual patients are illustrated in Figure 2. Maximal number of different presentations in MM for one patient was 25, maximal number of different sites presented for one patient in one MM was eight, and most of the patients are presented only for one site of infection.

Fig. 2

Description of redundant lines per patient in the database number of presentations in a multidisciplinary meeting (MM) per patient according to the type of referral centre: CRIOAc ("Centre de Référence pour les Infections Osteo-Articulaires Complexes") or associate centre (n = 17,748 individual patients).

Main characteristics of the patients

Between 2012 and 2016, the mean age of patients was 62.1 years (SD 18.4) and 10,961 (61.8%) were male. BJI was considered as complex for 19,355 cases (63.2%) and 13,376 PJIs were presented (43.7%). Missing data varied substantially according to the centres and types of variables. There were few missing data (less than 5%) for age, sex, site of infection, postoperative antibiotic therapy, definition of complexity, and unstructured abstract of patient history (except for one centre where missing data was 11.9%). There were less than 20% of missing data for type and side of infection. For surgical procedure and microbiology, mean percentages of missing data were 10.4% and 6.0%, respectively, but could reach 42.3% and 36.7% according to the centre. Some variables were poorly documented (mean percentage and SDs of missing data): weight (49.4% (SD 38.0%)), height (51.6% (SD 36.2%)), BMI (54.9% (SD 35.5%)), comorbidities (41.2% (SD 17.2%)), American Society of Anesthesiologists (ASA) score (54.9% (SD 39.0%)),¹⁰ creatinine (57.1% (SD 34.4%)), and CRP levels (58.4% (SD 29.1%)).

Quality assessment

The random sample of patients recorded in the registry over the 2014 to 2016 period was analyzed across 18 different centres, representing 283 patients. Crude mean data consistency and completeness were 86.5% and 72.6%, respectively. There were no significant differences between centres for consistency, but there was a high variability in completeness. Data consistency and completeness were stable over time (Figure 3). Considering missing data as negative data for categorical comorbidities, variables dramatically improved data consistency and completeness (88.2% and 88.9%, respectively) (Figure 4). Detailed analysis by variable is shown in Supplementary Figures a and b.

Fig. 3

Crude data consistency and completeness. a) Crude data consistency by referral centre. Each bar represents the percentage of consistent data by centre. The grey line represents the mean consistency. b) Crude data consistency by year. c) Crude data completeness by referral centre. The grey line represents the mean completeness. d) Crude data completeness by year.

Fig. 4

Data consistency and completeness, considering missing data as a negative result. a) Data consistency by referral centre. Each bar represents the percentage of consistent data by centre. The grey line represents the mean consistency. b) Data consistency by year. c) Data completeness by referral centre. The grey line represents the mean completeness. d) Data completeness by year.

Main results and strength

The French registry of CRIOAcs is one of the largest prospective databases on BJI. These infections, even more for complex cases, are difficult to investigate because of their relative rarity and heterogeneity.^1,11–13 This cohort is a unique opportunity to obtain specific epidemiological data based on a multicentric cohort, with a multidisciplinary approach, in a real-life setting, using an automated data extraction process. Despite not being initially designed for research, the quality assessment of the cleaned BJI national registry showed global rates of data consistency and completeness higher than 85%, allowing further use for epidemiological purposes.

Quality assessment

The quality of registry data is considered to be a fundamental element for ensuring better interpretation.¹⁴ However, few studies on data quality have been published in the area of orthopaedic registries.^15,16 An excellent completeness of case ascertainment is not a specific goal for a clinical registry, here relying only on referral centres. Conversely, internal completeness is crucial in interpreting the results. Considering the unfilled fields for the medical history in the eCRF as negative results, we obtained an internal completeness greater than 90% for the most essential items describing the patient presentation, and more than 80% for the MM recommendations (without analyzing unstructured data). These results can be considered as acceptable. On the contrary, the weight, height, and biological measurements as well as the ASA score did not seem reliable for analysis. This is detrimental to the comparability of our results as the ASA score and BMI are among the recommended items for arthroplasty registries.¹⁶ Consistency is considered as acceptable with a global percentage far greater than 80%,⁷ despite being lower for some critical items: type of infection and surgical recommendations. This is partly due to the structure of the data entries and the constitution of the controlled dataset; in the initial eCRF, the 'type of infection' item is structured as a list, associating the presence of an orthopaedic implant and the type of infection, and many fields are filled only for one of the two pieces of information. For surgical recommendations, the controlled dataset extracted from medical charts often recorded the surgical procedure actually performed rather than the one proposed. The consistency was heterogeneous depending on the considered variable. By excluding variables with unacceptable consistency or completeness, the global mean consistency achieved 92.9%, which can be considered as reliable. Assessing the good data quality from the French registry of C-BJIs was a preliminary step before providing a precise epidemiology analysis, and we believe that this procedure should be extended to the other registries to guarantee their reliability.

Other databases in the literature

One of the difficulties considering BJI epidemiology is due to the important heterogeneity of presentations in their pathophysiology, diagnosis, medical and/or surgical management, and outcomes. Other cohorts have been conducted worldwide for investigation of BJI, with different strategies in different countries. Most of these studies are focused on one type of infection, mostly PJIs, with the implementation of national registries following arthroplasty procedures worldwide.^17–21 Sub-analyses of these observational nationwide cohorts focus on PJI, with a quasi-exhaustive detection. However, the maintenance of these kinds of registries is costly and these cohorts are not designed for PJI description. They are fitted to describe risk factors and incidence of PJI, but critical data are lacking for a specific analysis of management.

Few registries have been dedicated specifically to BJI and data are mainly issued from retrospective cohorts. However, some collaborative multicentric cohorts²² or pooled meta-analyses on individual patients²³ have been performed specifically for PJI at a national or international scale. They provided a more precise description of microbiological and clinical aspects of PJI. However, the heterogeneity of records and the frequent retrospective design limit their interpretation.

The originality of the French registry is to combine the systematic and exhaustive record of unselected patients with BJI presented in referral centres, with a detailed and quite homogeneous description of BJI. Furthermore, not only PJIs are represented, and despite their heterogeneity it will be possible to analyze the other types of BJI precisely.

Main limitations

However, this study raised some limitations about the analysis of the registry. Firstly, we found a substantial heterogeneity in the database filling. In fact, many variables were coded quite differently according to the centre. Some variables such as medical history or detailed microbiology were sometimes filled like structured data, and sometimes detailed in the form of unstructured data, complicating their analysis. A basic semi-automatic text analysis was necessary to extract and combine information from both data sources. Some centres constantly unfill some variables such as height and weight or CRP measures, thus their interpretation was consequently biased. More generally, the rate of missing data was highly variable depending on centres and variables. This fact is partly due to the structure of the standardized case report form, where data are often recorded only if present (negative results or absence of symptom not recorded). However, considering missing data as negative results for categorical variables of medical history in a sensitivity analysis substantially allowed the improvement of data completeness and data consistency. While this attitude might seem questionable, these data were artificially missing due to the structure of the eCRF and the reorganization of the database for analysis purposes, going from a list of different items to several independent variables. Thus, we controlled the two hypotheses: 1) if not filled, the data are missing; and 2) if not filled, the data are negative. The quality parameters being better by considering as negative the variables not filled, we supposed that the item not checked in the list could be regarded as absent.

Unfortunately, five referral centres were not able to provide a controlled dataset. Their data quality was consequently not addressed. This analysis will be necessary before the inclusion of patients from these centres in further epidemiological studies.

It eventually became apparent that the main limitation of the registry was the absence of recorded follow-up data for individual patients. Thus, evaluation of the therapeutic options will need a retrospective analysis of patient records.

Potential ways of improvement

We highlighted in this study the matter of unstructured data analysis. Indeed, many missing data were encoded as commentaries and were not directly analyzable. The use of natural language processing (NLP) will help to improve data quality.^24–26 Results of quality analysis for both variables extracted by NLP are heterogeneous. For cognitive impairment, the consistency is highly acceptable in all centres. Indeed, this clinical parameter is really important for the decision-making process and is specified in the comments when present. Conversely, the results are more heterogeneous for excessive alcohol consumption, and this variable is one of those which should be analyzed with caution.

Another way to improve the quality of the French registry will be to harmonize the database filling. In this perspective, a common reflection ('think tank') on an adapted case report form more accurate for research was submitted to the national scientific council for CRIOAc, and specific formations for database filling with a better standardization will be developed. In order to assess the efficiency of these measures, regular quality analysis should be planned.

Finally, the implementation of a long-term follow-up after presentation at MM could constitute a substantial improvement of the registry. As a first step, follow-up data for a pre-specified BJI will be collected retrospectively in selected referral centres to assess its feasibility. Secondly, systematic recording of long-term monitoring data could be set up for some test centres, for preliminary analysis of acceptability and cost. Another method of improvement could be the linkage of the registry with the Hospital Discharge Database and National Health Insurance Database by the patient's unique health insurance number. This database evolution would allow the detection of late re-hospitalizations and mortality.

Potential utilizations

In conclusion, the French registry of C-BJIs offers acceptable quality parameters and a sufficient size to accurately describe the epidemiology of C-BJI in France over time. The implementation of long-term outcome data could facilitate the evaluation of therapeutic strategies of these infrequent diseases. A scientific council composed of eight members representative of each CRIOAc, of each region of France, and of each speciality (orthopaedic surgery, infectious diseases, and microbiology) was created to centralize research projects and perform feasibility analyses for potential clinical trials.