Skip to main content

Advertisement

Log in

Assessing and mitigating the effects of class imbalance in machine learning with application to X-ray imaging

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Machine learning (ML) algorithms are well known to exhibit variations in prediction accuracy when provided with imbalanced training sets typically seen in medical imaging (MI) due to the imbalanced ratio of pathological and normal cases. This paper presents a thorough investigation of the effects of class imbalance and methods for mitigating class imbalance in ML algorithms applied to MI.

Methods

We first selected five classes from the Image Retrieval in Medical Applications (IRMA) dataset, performed multiclass classification using the random forest model (RFM), and then performed binary classification using convolutional neural network (CNN) on a chest X-ray dataset. An imbalanced class was created in the training set by varying the number of images in that class. Methods tested to mitigate class imbalance included oversampling, undersampling, and changing class weights of the RFM. Model performance was assessed by overall classification accuracy, overall F1 score, and specificity, recall, and precision of the imbalanced class.

Results

A close-to-balanced training set resulted in the best model performance, and a large imbalance with overrepresentation was more detrimental to model performance than underrepresentation. Oversampling and undersampling methods were both effective in mitigating class imbalance, and efficacy of oversampling techniques was class specific.

Conclusion

This study systematically demonstrates the effect of class imbalance on two public X-ray datasets on RFM and CNN, making these findings widely applicable as a reference. Furthermore, the methods employed here can guide researchers in assessing and addressing the effects of class imbalance, while considering the data-specific characteristics to optimize imbalance mitigating methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Availability of data and material

Datasets used in this study include an X-ray image dataset (IRMA), a handwritten digits image dataset (MNIST), and a chest X-Ray binary dataset consisting of pneumonia and normal images. All are publicly available.

References

  1. Chan S, Siegel EL (2019) Will machine learning end the viability of radiology as a thriving medical specialty? Br J Radiol 92:20180416. https://doi.org/10.1259/bjr.20180416

    Article  PubMed  Google Scholar 

  2. Chen C, Liaw A, Brieman L (2004) Using random forest to learn imbalanced data: Technical Report No. 666. University of California, Berkley. Using Random Forest to Learn Imbalanced Data

  3. Barandela R, Sánchez JS, García V, Rangel E (2003) Strategies for learning in class imbalance problems. Pattern Recognit 36:849–851. https://doi.org/10.1016/S0031-3203(02)00257-1

    Article  Google Scholar 

  4. Klement W, Wilk S, Michalowski W, Matwin S (2011) Classifying severely imbalanced data

  5. Tang A, Tam R, Cadrin-Chênevert A, Guest W, Chong J, Barfett J, Chepelev L, Cairns R, Mitchell JR, Cicero MD, Poudrette MG, Jaremko JL, Reinhold C, Gallix B, Gray B, Geis R, O’Connell T, Babyn P, Koff D, Ferguson D, Derkatch S, Bilbily A, Shabana W (2018) Canadian association of radiologists white paper on artificial intelligence in radiology. Can Assoc Radiol J 69:120–135

    Article  Google Scholar 

  6. Balki I, Amirabadi A, Levman J, Martel AL, Emersic Z, Meden B, Garcia-Pedrero A, Ramirez SC, Kong D, Moody AR, Tyrrell PN (2019) Sample-size determination methodologies for machine learning in medical imaging research: a systematic review. Can Assoc Radiol J 70:344–353

    Article  Google Scholar 

  7. Lehmann T, Fischer B, Güld M, Thies C, Keysers D, Deselaers T, Schubert H, Wein B, Spitzer K (2004) The IRMA reference database and its use for content-based image retrieval in medical applications. Science 5:3–6

    Google Scholar 

  8. Kermany DS, Goldbaum M, Cai W, Valentim CCS, Liang H, Baxter SL, McKeown A, Yang G, Wu X, Yan F, Dong J, Prasadha MK, Pei J, Ting M, Zhu J, Li C, Hewett S, Dong J, Ziyar I, Shi A, Zhang R, Zheng L, Hou R, Shi W, Fu X, Duan Y, Huu VAN, Wen C, Zhang ED, Zhang CL, Li O, Wang X, Singer MA, Sun X, Xu J, Tafreshi A, Lewis MA, Xia H, Zhang K (2018) Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172:1122–1131.e9. https://doi.org/10.1016/j.cell.2018.02.010

    Article  CAS  PubMed  Google Scholar 

  9. Cui Z, Gong G (2018) The effect of machine learning regression algorithms and sample size on individualized behavioral prediction with functional connectivity features. Neuroimage 178:622–637. https://doi.org/10.1016/j.neuroimage.2018.06.001

    Article  PubMed  Google Scholar 

  10. Cohen O, Zhu B, Rosen MS (2018) MR fingerprinting deep reconstruction network (DRONE). Magn Reson Med 80:885–894. https://doi.org/10.1002/mrm.27198

    Article  PubMed  PubMed Central  Google Scholar 

  11. Blagus R, Lusa L (2010) Class prediction for high-dimensional class-imbalanced data. BMC Bioinformatics 11:523. https://doi.org/10.1186/1471-2105-11-523

    Article  PubMed  PubMed Central  Google Scholar 

  12. Japkowicz N, Stephen S (2002) The class imbalance problem: a systematic study. Intell Data Anal 6:429–449. https://doi.org/10.3233/ida-2002-6504

    Article  Google Scholar 

  13. Park SH, Han K (2018) Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology 286:800–809. https://doi.org/10.1148/radiol.2017171920

    Article  Google Scholar 

  14. Chawla NV (2006) Data mining for imbalanced datasets: an overview. In: Data mining and knowledge discovery handbook. Springer, pp 853–867

  15. Abd Elrahman SM, Abraham A (2013) A review of class imbalance problem. J Netw Innov Comput 1:332–340

    Google Scholar 

  16. Hitzl W, Reitsamer HA, Hornykewycz K, Mistlberger A, Grabner G (2003) Application of discriminant, classification tree and neural network analysis to differentiate between potential glaucoma suspects with and without visual field defects. J Theor Med 5:161–170. https://doi.org/10.1080/10273360410001728011

    Article  Google Scholar 

Download references

Funding

No funding was received for this work.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Pascal N. Tyrrell.

Ethics declarations

Conflict of interest

All authors have declared that they have no conflict of interest.

Code availability

All code was custom using Python 3.7.3 and included in Supplementary Materials (Online Resources 2–9 and 11).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qu, W., Balki, I., Mendez, M. et al. Assessing and mitigating the effects of class imbalance in machine learning with application to X-ray imaging. Int J CARS 15, 2041–2048 (2020). https://doi.org/10.1007/s11548-020-02260-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-020-02260-6

Keywords

Navigation