Key indicators of phase transition for clinical trials through machine learning

Highlights

• Protocol features across therapeutic areas and trial phases linked to phase success.

• Supervised machine learning predicts drug transition across clinical trial phases.

• Clinical trials phase transitions predicted with an average accuracy of 80%.

• Natural language algorithms to study eligibility criteria role on phase success.

• Updated estimates for phase success and likelihood of approval.

A significant number of drugs fail during the clinical testing stage. To understand the attrition of drugs through the regulatory process, here we review and advance machine-learning (ML) and natural language-processing algorithms to investigate the importance of factors in clinical trials that are linked with failure in Phases II and III. We find that clinical trial phase transitions can be predicted with an average accuracy of 80%. Identifying these trials provides information to sponsors facing difficult decisions about whether these higher risk trials should be modified or halted. We also find common protocol characteristics across therapeutic areas that are linked to phase success, including the number of endpoints and the complexity of the eligibility criteria.

Introduction

High attrition in the drug development pipeline is well documented in the literature 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17. The most critical failures happen at later stages; ∼60–70% of Phase II trials and 30–40% of Phase III trials are unsuccessful; meaning that 60–70% of the drugs that make it to Phase II will not transition to Phase III. Similarly, of the drugs that make it to Phase III, 30–40% will not transition to New Drug Application (NDA)/Biologics License Application (BLA) submission 2, 4, 8. Overall, ∼11–19% of drugs in testing will make it from Phase I to final regulatory approval for a lead indication 2, 4, 8.

The biomedical research enterprise has made substantial technological and scientific advancements over the past few decades, yet, overall it is spending more on research and development (R&D) without seeing a comparable increase in novel therapeutics, as assessed by new molecular entities (NMEs) and new biologic entities (NBEs) or by life expectancy gains 3, 5, 6, 18, 19. Given the significant human and financial costs associated with bringing a drug to market 3, 8, 10, with estimates varying between US$600 million and US$2.8 billion, most stakeholders agree that the attrition rate is unsustainably high 2, 3, 4, 8, 18, 20, 21.

In an attempt to improve productivity and reduce R&D costs, numerous researchers and other stakeholders have tried to untangle the reasons for attrition by examining various molecular, strategic, financial, and regulatory factors. At a very basic level, researchers have found that most drugs fail because of lack of efficacy followed by safety issues 9, 11, 22; however, a large body of literature provides more detail and insight into the factors that potentially affect trial performance. Several studies consider the characteristics of the drug itself. For example, NMEs are more likely to fail compared with other drugs 8, 20, and biologics (or large molecules) have a higher success rate than small molecules in clinical trials 4, 6. Some research questions whether the target-based approach and/or drug-likeness approaches utilized earlier in the drug discovery process have led to more drugs entering the pipeline that later fail because of safety issues 6, 23.

Additional research looks beyond the drug itself and points to higher failure rates in trials for specific diseases and disorders, including chronic diseases such as Alzheimer’s and diabetes, whereas others point to higher failure across entire therapeutic areas, such as oncology, infectious disease, and central nervous system disorders 4, 5, 8, 11, 20. Further studies indicate that strategic factors might also have a role. For example, self-originated drugs fail at higher rates than those that are licensed-in 2, 4. Also, research shows that pharmaceutical companies have increasingly invested in less commercially crowded areas, such as immunomodulatory drugs, as well as in drugs with novel mechanisms of action, where there might be not only higher expected revenues, but also higher failure rates 2, 5, 6.

Some research indicates that trial protocol complexity, longer cycle times, and increased investigative site work burden also contribute to poor trial performance and failure 22, 24, 25. The type of company making strategic R&D decisions also appears to matter, because smaller companies are more likely to experience failure compared with larger companies; this is true whether company size is measured by pharmaceutical sales or by R&D budget 2, 8, 24. Finally, some research posits that barriers erected by regulatory agencies contribute to trial failure, although drugs with newer special designations, such as orphan status, might in fact improve success rates 6, 8.

In addition to this complex body of literature, recent research has attempted to use larger data sets 16, 26 and ML methodologies 24, 27, 28, 29, 30, 31, 32, 33 to analyze failures in greater detail and consider additional factors that might be contributing to pipeline performance. Some studies that have developed ML models 24, 27 present similar analyses to those presented here. However, they have focused on answering different questions. For instance, DiMasi et al. [24] developed a model to predict regulatory approval after Phase II for oncology trials using 98 oncology drugs from the top 50 pharmaceutical companies. They used logistic regression and ML methods [Random Forests (RF), particularly for variable selection purposes] to predict and identify the most important individual factors associated with regulatory approval. A similar study was done by Lo et al. [27], who, similar to DiMasi et al., analyzed the data of completed (finished) trials to predict drug approvals (probability of success) and variable importance using drug-development and clinical-trial data from 2003 to 2015. The analysis was performed by developing a set of distinct ML models, including RF, neural networks, and decisions trees, among others. Indeed, they found that k-nearest neighbor (k-NN) imputation with RF provided the best predictive performance. They also identified trial outcomes, trial status, and trial accrual rates as significant factors for prediction.

Given the extensive work using ML methods to investigate factors impacting pipeline performance, we similarly investigate factors before a trial begins that can explain trial outcome (defined as success or failure) and phase transition. Two of the most comprehensive data sets of clinical trial designs and drug characteristics available (the Aggregate Analysis of ClinicalTrials.gov, AACT [34], based on https://clinicaltrials.gov/ and Biomedtracker [26] data sets) were used for this case study. A series of supervised ML (SML) models, which are informed with clinical trial data, are used to forecast whether a drug will successfully transition from the beginning of one phase to next (e.g., if a drug entering Phase II will transition to III and Phase III to NDA/BLA submission or regulatory approval), and to identify which factors, if any, were associated with the chances of a drug advancing toward regulatory approval. A thorough literature search is then presented, which seeks to provide a better understanding of the importance of such factors.

The case study evaluates the capability of ML models to predict phase success and failure. Dozens of characteristics from individual clinical trials are used, including the design of the trial protocol (e.g., number of endpoints or number of study arms), operational characteristics (e.g., number of countries where trial is being conducted or stakeholder type serving as lead sponsor), and other potential covariates, such as the target enrollment of each trial (see S1 in the supplemental material online for the complete list of covariates). Several characteristics not utilized by previous studies are explored, including eligibility criteria complexity. Eligibility criteria and a significant number of other clinical trial characteristics exist only as unstructured and free text data in the available data sets. Therefore, to analyze the importance of eligibility criteria in trial success, a natural language processing (NLP) algorithm (based on pattern matching and replacement and character vector count) was used to generate a metric for eligibility criteria complexity (see Methods and supplemental material online for developmental details). Although NLP techniques have been widely used in systems biology [35], ontology for drug discovery [36] and in biomedical context 37, 38, 39, such as in drug name recognition in biomedical text [40], this case study is the first to use NLP to create and explore a metric for eligibility criteria complexity and to recognize that complexity as an important factor in trial outcome.

The SML models used in this study can identify the outcome of a trial phase with an average accuracy of 80% when analyzing specific therapeutic areas. Access to a model with this level of predictive accuracy offers insight in two distinct capacities. From a trialist’s perspective, the model can be used as an iterative tool offering counsel in the protocol design and operational characteristics that should be pursued, and those that should be avoided or altered. From a commercial perspective, the predictive capabilities of the model provide more information for decision-making when assessing portfolios and allocating resources.