The Trade Secret Taboo: Open Science Methods are Required to Improve Prediction Models in Sports Medicine and Performance

Abstract

Clinical prediction models in sports medicine that utilize regression or machine learning techniques have become more widely published, used, and disseminated. However, these models are typically characterized by poor methodology and incomplete reporting, and an inadequate evaluation of performance, leading to unreliable predictions and weak clinical utility within their intended sport population. Before implementation in practice, models require a thorough evaluation. Strong replicable methods and transparency reporting allow practitioners and researchers to make independent judgments as to the model’s validity, performance, clinical usefulness, and confidence it will do no harm. However, this is not reflected in the sports medicine literature. As shown in a recent systematic review of models for predicting sports injury models, most were typically characterized by poor methodology, incomplete reporting, and inadequate performance evaluation. Because of constraints imposed by data from individual teams, the development of accurate, reliable, and useful models is highly reliant on external validation. However, a barrier to collaboration is a desire to maintain a competitive advantage; a team’s proprietary information is often perceived as high value, and so these ‘trade secrets’ are frequently guarded. These ‘trade secrets’ also apply to commercially available models, as developers are unwilling to share proprietary (and potentially profitable) development and validation information. In this Current Opinion, we: (1) argue that open science is essential for improving sport prediction models and (2) critically examine sport prediction models for open science practices.

Appendix 1: TRIPOD Checklist: Prediction Model Development and Validation

Section/topic	Item		Checklist item	Page
Title and abstract
Title	1	D;V	Identify the study as developing and/or validating a multivariable prediction model, the target population, and the outcome to be predicted
Abstract	2	D;V	Provide a summary of objectives, study design, setting, participants, sample size, predictors, outcome, statistical analysis, results, and conclusions
Introduction
Background and objectives	3a	D;V	Explain the medical context (including whether diagnostic or prognostic) and rationale for developing or validating the multivariable prediction model, including references to existing models
	3b	D;V	Specify the objectives, including whether the study describes the development or validation of the model or both
Methods
Source of data	4a	D;V	Describe the study design or source of data (e.g., randomized trial, cohort, or registry data), separately for the development and validation data sets, if applicable
	4b	D;V	Specify the key study dates, including start of accrual; end of accrual; and, if applicable, end of follow-up
Participants	5a	D;V	Specify key elements of the study setting (e.g., primary care, secondary care, general population) including number and location of centers
	5b	D;V	Describe eligibility criteria for participants
	5c	D;V	Give details of treatments received, if relevant
Outcome	6a	D;V	Clearly define the outcome that is predicted by the prediction model, including how and when assessed
	6b	D;V	Report any actions to blind assessment of the outcome to be predicted
Predictors	7a	D;V	Clearly define all predictors used in developing or validating the multivariable prediction model, including how and when they were measured
	7b	D;V	Report any actions to blind assessment of predictors for the outcome and other predictors
Sample size	8	D;V	Explain how the study size was arrived at
Missing data	9	D;V	Describe how missing data were handled (e.g., complete-case analysis, single imputation, multiple imputation) with details of any imputation method
Statistical analysis methods	10a	D	Describe how predictors were handled in the analyses
	10b	D	Specify type of model, all model-building procedures (including any predictor selection), and method for internal validation
	10c	V	For validation, describe how the predictions were calculated
	10d	D;V	Specify all measures used to assess model performance and, if relevant, to compare multiple models
	10e	V	Describe any model updating (e.g., recalibration) arising from the validation, if done
Risk groups	11	D;V	Provide details on how risk groups were created, if done
Development vs validation	12	V	For validation, identify any differences from the development data in setting, eligibility criteria, outcome, and predictors
Results
Participants	13a	D;V	Describe the flow of participants through the study, including the number of participants with and without the outcome and, if applicable, a summary of the follow-up time. A diagram may be helpful
	13b	D;V	Describe the characteristics of the participants (basic demographics, clinical features, available predictors), including the number of participants with missing data for predictors and outcome
	13c	V	For validation, show a comparison with the development data of the distribution of important variables (demographics, predictors and outcome)
Model development	14a	D	Specify the number of participants and outcome events in each analysis
	14b	D	If done, report the unadjusted association between each candidate predictor and outcome
Model specification	15a	D	Present the full prediction model to allow predictions for individuals (i.e., all regression coefficients, and model intercept or baseline survival at a given time point)
	15b	D	Explain how to the use the prediction model
Model performance	16	D;V	Report performance measures (with CIs) for the prediction model
Model updating	17	V	If done, report the results from any model updating (i.e., model specification, model performance)
Discussion
Limitations	18	D;V	Discuss any limitations of the study (such as nonrepresentative sample, few events per predictor, missing data)
Interpretation	19a	V	For validation, discuss the results with reference to performance in the development data, and any other validation data
	19b	D;V	Give an overall interpretation of the results, considering objectives, limitations, results from similar studies, and other relevant evidence
Implications	20	D;V	Discuss the potential clinical use of the model and implications for future research
Other information
Supplementary information	21	D;V	Provide information about the availability of supplementary resources, such as study protocol, Web calculator, and data sets
Funding	22	D;V	Give the source of funding and the role of the funders for the present study

1. CIs confidence intervals, TRIPOD Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis
2. Items relevant only to the development of a prediction model are denoted by D, items relating solely to a validation of a prediction model are denoted by V, and items relating to both are denoted D;V. We recommend using the TRIPOD Checklist in conjunction with the TRIPOD Explanation and Elaboration document