Skip to main content
Log in

Generating Longitudinal Atrophy Evaluation Datasets on Brain Magnetic Resonance Images Using Convolutional Neural Networks and Segmentation Priors

  • Original Article
  • Published:
Neuroinformatics Aims and scope Submit manuscript

Abstract

Brain atrophy quantification plays a fundamental role in neuroinformatics since it permits studying brain development and neurological disorders. However, the lack of a ground truth prevents testing the accuracy of longitudinal atrophy quantification methods. We propose a deep learning framework to generate longitudinal datasets by deforming T1-w brain magnetic resonance imaging scans as requested through segmentation maps. Our proposal incorporates a cascaded multi-path U-Net optimised with a multi-objective loss which allows its paths to generate different brain regions accurately. We provided our model with baseline scans and real follow-up segmentation maps from two longitudinal datasets, ADNI and OASIS, and observed that our framework could produce synthetic follow-up scans that matched the real ones (Total scans= 584; Median absolute error: 0.03 ± 0.02; Structural similarity index: 0.98 ± 0.02; Dice similarity coefficient: 0.95 ± 0.02; Percentage of brain volume change: 0.24 ± 0.16; Jacobian integration: 1.13 ± 0.05). Compared to two relevant works generating brain lesions using U-Nets and conditional generative adversarial networks (CGAN), our proposal outperformed them significantly in most cases (p < 0.01), except in the delineation of brain edges where the CGAN took the lead (Jacobian integration: Ours - 1.13 ± 0.05 vs CGAN - 1.00 ± 0.02; p < 0.01). We examined whether changes induced with our framework were detected by FAST, SPM, SIENA, SIENAX, and the Jacobian integration method. We observed that induced and detected changes were highly correlated (Adj. R2 > 0.86). Our preliminary results on harmonised datasets showed the potential of our framework to be applied to various data collections without further adjustment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. See https://surfer.nmr.mgh.harvard.edu/fswiki/LongitudinalData

  2. Available at http://web.iyte.edu.tr/~bilgekaracali/VoxelVolumeMatching.tar.gz

  3. Available at https://www.nitrc.org/projects/image_synthesis/

  4. Available at https://github.com/agis85/multimodal_brain_synthesis.

  5. Available at https://github.com/NIC-VICOROB/MS_Lesions_Generator

  6. Available at https://github.com/phillipi/pix2pix

  7. adni.loni.usc.edu The ADNI was launched in 2003 as a public-private partnership, led by Principal Investigator Michael W. Weiner, MD. The primary goal of ADNI has been to test whether serial magnetic resonance imaging (MRI), positron emission tomography (PET), other biological markers, and clinical and neuropsychological assessment can be combined to measure the progression of mild cognitive impairment (MCI) and early Alzheimer’s disease (AD). For up-to-date information, see www.adni-info.org.

References

  • Amiri, H., de Sitter, A., Bendfeldt, K., Battaglini, M., Wheeler-Kingshott, C.A.G., Calabrese, M., Geurts, J.J., Rocca, M.A., Sastre-Garriga, J., Enzinger, C., & et al. (2018). Urgent challenges in quantification and interpretation of brain grey matter atrophy in individual MS patients using MRI. NeuroImage: Clinical, 19, 466–475.

    Article  Google Scholar 

  • Andersson, J.L., Jenkinson, M., Smith, S., & et al. (2007). Non-linear registration aka Spatial normalisation FMRIB Technial Report TR07JA2. FMRIB Analysis Group of the University of Oxford.

  • Ashburner, J., Barnes, G., & Chen, C. (2012). SPM12 Manual. www.fil.ion.ucl.ac.uk (Online; Accessed 21 Jun 2018.

  • Battaglini, M., Jenkinson, M., De Stefano, N., & Initiative, A.D.N. (2018). SIENA-XL for improving the assessment of gray and white matter volume changes on brain MRI. Human Brain Mapping, 39(3), 1063–1077.

    Article  PubMed  Google Scholar 

  • Bernal, J., Kushibar, K., Asfaw, D.S., Valverde, S., Oliver, A., Martí, R., & Lladó, X. (2019a). Deep convolutional neural networks for brain image analysis on magnetic resonance imaging: a review. Artificial Intelligence in Medicine, 95, 64–81.

    Article  PubMed  Google Scholar 

  • Bernal, J., Kushibar, K., Cabezas, M., Valverde, S., Oliver, A., & Lladó, X. (2019b). Quantitative analysis of patch-based fully convolutional neural networks for tissue segmentation on brain magnetic resonance imaging. IEEE Access, 7, 89986–90002.

    Article  Google Scholar 

  • Brosch, T., Tang, L.Y., Yoo, Y., Li, D.K., Traboulsee, A., & Tam, R. (2016). Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation. IEEE Transactions on Medical Imaging, 35(5), 1229–1239.

    Article  PubMed  Google Scholar 

  • Chartsias, A., Joyce, T., Giuffrida, M.V., & Tsaftaris, S.A. (2018). Multimodal MR synthesis via modality-invariant latent representation. IEEE Transactions on Medical Imaging, 37(3), 803–814.

    Article  PubMed  Google Scholar 

  • Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., & Ronneberger, O. (2016). 3D U-Net: learning dense volumetric segmentation from sparse annotation. In International conference on medical image computing and computer-assisted intervention (pp. 424–432): Springer.

  • Costa, P., Galdran, A., Meyer, M.I., Niemeijer, M., Abràmoff, M., Mendonċa, A.M., & Campilho, A. (2018). End-to-end adversarial retinal image synthesis. IEEE Transactions on Medical Imaging, 37 (3), 781–791.

    Article  PubMed  Google Scholar 

  • Cover, K.S., van Schijndel, R.A., van Dijk, B.W., Redolfi, A., Knol, D.L., Frisoni, G.B., Barkhof, F., Vrenken, H., Initiative, A.D.N., & et al. (2011). Assessing the reproducibility of the SIENAX and SIENA brain atrophy measures using the ADNI back-to-back MP-RAGE MRI scans. Psychiatry Research: Neuroimaging, 193(3), 182–190.

    Article  PubMed  Google Scholar 

  • Crum, W.R., Camara, O., & Hill, D.L.G. (2006). Generalized overlap measures for evaluation and validation in medical image analysis. IEEE Transactions on Medical Imaging, 25(11), 1451– 1461.

    Article  PubMed  Google Scholar 

  • de Boer, R., Vrooman, H.A., Ikram, M.A., Vernooij, M.W., Breteler, M.M., van der Lugt, A., & Niessen, W.J. (2010). Accuracy and reproducibility study of automatic MRI brain tissue segmentation methods. NeuroImage, 51(3), 1047–1056.

    Article  PubMed  Google Scholar 

  • Dice, L.R. (1945). Measures of the amount of ecologic association between species. Ecology, 26 (3), 297–302.

    Article  Google Scholar 

  • Ens, K., Wenzel, F., Young, S., Modersitzki, J., & Fischer, B. (2009). Design of a synthetic database for the validation of non-linear registration and segmentation of magnetic resonance brain images. In Medical imaging 2009: image processing, (Vol. 7259 p. 725933). International Society for Optics and Photonics.

  • Filippi, M., Rocca, M.A., Ciccarelli, O., De Stefano, N., Evangelou, N., Kappos, L., Rovira, A., Sastre-Garriga, J., Tintorè, M., Frederiksen, J.L., Gasperini, C., Palace, J., Reich, D.S., Banwell, B., Montalban, X., & Barkhof, F. (2016). MRI criteria for the diagnosis of multiple sclerosis: MAGNIMS consensus guidelines. The Lancet Neurology, 15(3), 292–303.

    Article  PubMed  PubMed Central  Google Scholar 

  • Fischl, B., Salat, D.H., Busa, E., Albert, M., Dieterich, M., Haselgrove, C., Van Der Kouwe, A., Killiany, R., Kennedy, D., Klaveness, S., & et al. (2002). Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain. Neuron, 33(3), 341–355.

    Article  CAS  PubMed  Google Scholar 

  • Fox, N.C., Jenkins, R., Leary, S.M., Stevenson, V.L., Losseff, N.A., Crum, W.R., Harvey, R.J., Rossor, M.N., Miller, D.H., & Thompson, A.J. (2000). Progressive cerebral atrophy in MS: a serial study using registered, volumetric MRI. Neurology, 54(4), 807–812.

    Article  CAS  PubMed  Google Scholar 

  • Freeborough, P.A., & Fox, N.C. (1997). The boundary shift integral: an accurate and robust measure of cerebral volume changes from registered repeat MRI. IEEE Transactions on Medical Imaging, 16(5), 623–629.

    Article  CAS  PubMed  Google Scholar 

  • Frid-Adar, M., Diamant, I., Klang, E., Amitai, M., Goldberger, J., & Greenspan, H. (2018). GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification. Neurocomputing, 321, 321–331.

    Article  Google Scholar 

  • Ghafoorian, M., Karssemeijer, N., Heskes, T., Uden, I.W., Sanchez, C.I., Litjens, G., Leeuw, F.E., Ginneken, B., Marchiori, E., & Platel, B. (2017). Location sensitive deep convolutional neural networks for segmentation of white matter hyperintensities. Scientific Reports, 7(1), 5110.

    Article  PubMed  PubMed Central  Google Scholar 

  • Glorot, X., Bordes, A., & Bengio, Y. (2011). Deep sparse rectifier neural networks. In Proceedings of the 14th international conference on artificial intelligence and statistics (pp. 315–323).

  • Guerrero, R., Qin, C., Oktay, O., Bowles, C., Chen, L., Joules, R., Wolz, R., Valdés-Hernández, M., Dickie, D., Wardlaw, J., & et al. (2018). White matter hyperintensity and stroke lesion segmentation and differentiation using convolutional neural networks. NeuroImage: Clinical, 17, 918–934.

    Article  CAS  Google Scholar 

  • Haijma, S.V., Van Haren, N., Cahn, W., Koolschijn, P.C.M., Hulshoff Pol, H.E., & Kahn, R.S. (2012). Brain volumes in schizophrenia: a meta-analysis in over 18 000 subjects. Schizophrenia Bulletin, 39(5), 1129–1138.

    Article  PubMed  PubMed Central  Google Scholar 

  • He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision (pp. 1026–1034).

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778).

  • Heimann, T., Van Ginneken, B., Styner, M.A., Arzhaeva, Y., Aurich, V., Bauer, C., Beck, A., Becker, C., Beichel, R., Bekes, G., & et al. (2009). Comparison and evaluation of methods for liver segmentation from CT datasets. IEEE Transactions on Medical Imaging, 28(8), 1251–1265.

    Article  PubMed  Google Scholar 

  • Hore, A., & Ziou, D. (2010). Image quality metrics: PSNR vs. SSIM. In Proceedings of the 20th international conference on pattern recognition (pp. 2366–2369).

  • Iglesias, J.E., Liu, C.Y., Thompson, P.M., & Tu, Z. (2011). Robust brain extraction across datasets and comparison with publicly available methods. IEEE Transactions on Medical Imaging, 30(9), 1617–1634.

    Article  PubMed  Google Scholar 

  • Isola, P., Zhu, J.Y., Zhou, T., & Efros, A.A. (2017). Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1125–1134).

  • Jenkinson, M., & Smith, S. (2001). A global optimisation method for robust affine registration of brain images. Medical Image Analysis, 5(2), 143–156.

    Article  CAS  PubMed  Google Scholar 

  • Jenkinson, M., Bannister, P., Brady, M., & Smith, S. (2002). Improved optimization for the robust and accurate linear registration and motion correction of brain images. NeuroImage, 17(2), 825–841.

    Article  PubMed  Google Scholar 

  • Jia, G., Heymsfield, S.B., Zhou, J., Yang, G., & Takayama, Y. (2016). Quantitative biomedical imaging: techniques and clinical applications. BioMed Research International.

  • Karaçali, B., & Davatzikos, C. (2006). Simulation of tissue atrophy using a topology preserving transformation model. IEEE Transactions on Medical Imaging, 25(5), 649–652.

    Article  PubMed  Google Scholar 

  • Khanal, B., Ayache, N., & Pennec, X. (2017). Simulating longitudinal brain mris with known volume changes and realistic variations in image intensity. Frontiers in Neuroscience, 11, 132.

    Article  PubMed  PubMed Central  Google Scholar 

  • Kingma, D.P., & Ba, J. (2014). Adam: a method for stochastic optimization. coRR arXiv:1409.1556.

  • Krebs, J., e Delingette, H., Mailhé, B., Ayache, N., & Mansi, T. (2019). Learning a probabilistic model for diffeomorphic registration. IEEE Transactions on Medical Imaging, 38(9), 2165–2176.

    Article  PubMed  Google Scholar 

  • Li, G. (1985). Robust regression. Exploring Data Tables, Trends, and Shapes, 281, U340.

    Google Scholar 

  • Lin, M., Chen, Q., & Yan, S. (2013). Network in network. CoRR arXiv:1312.4400, pp. 1–10.

  • Marcus, D.S., Fotenos, A.F., Csernansky, J.G., Morris, J.C., & Buckner, R.L. (2010). Open access series of imaging studies: longitudinal MRI data in nondemented and demented older adults. Journal of Cognitive Neuroscience, 22(12), 2677–2684.

    Article  PubMed  PubMed Central  Google Scholar 

  • Nakamura, K., Guizard, N., Fonov, V.S., Narayanan, S., Collins, D.L., & Arnold, D.L. (2014). Jacobian integration method increases the statistical power to measure gray matter atrophy in multiple sclerosis. NeuroImage: Clinical, 4, 10–17.

    Article  Google Scholar 

  • Nakamura, K., Eskildsen, S.F., Narayanan, S., Arnold, D.L., Collins, D.L., Initiative, A.D.N., & et al. (2018). Improving the SIENA performance using BEaST brain extraction. PloS One, 13(9), e0196945.

    Article  PubMed  PubMed Central  Google Scholar 

  • Nyúl, L.G., Udupa, J.K., & Zhang, X. (2000). New variants of a method of MRI scale standardization. IEEE Transactions on Medical Imaging, 19(2), 143–150.

    Article  PubMed  Google Scholar 

  • Patenaude, B., Smith, S.M., Kennedy, D.N., & Jenkinson, M. (2011). A Bayesian model of shape and appearance for subcortical brain segmentation. NeuroImage, 56(3), 907–922.

    Article  PubMed  Google Scholar 

  • Rocca, M.A., Battaglini, M., Benedict, R.H., De Stefano, N., Geurts, J.J., Henry, R.G., Horsfield, M.A., Jenkinson, M., Pagani, E., & Filippi, M. (2017). Brain MRI atrophy quantification in MS: from methods to clinical application. Neurology, 88(4), 403–413.

    Article  PubMed  PubMed Central  Google Scholar 

  • Rovira, À., Wattjes, M.P., Tintoré, M., Tur, C., Yousry, T.A., Sormani, M.P., De Stefano, N., Filippi, M., Auger, C., Rocca, M.A., & et al. (2015). Evidence-based guidelines: MAGNIMS consensus guidelines on the use of MRI in multiple sclerosis—clinical implementation in the diagnostic process. Nature Reviews Neurology, 11(8), 471–482.

    Article  PubMed  Google Scholar 

  • Roy, S., Carass, A., & Prince, J. (2013). Magnetic resonance image example-based contrast synthesis. IEEE Transactions on Medical Imaging, 32(12), 2348–2363.

    Article  PubMed  PubMed Central  Google Scholar 

  • Rudick, R.A., Fisher, E., Lee, J-C, Simon, J., Jacobs, L., Multiple Sclerosis Collaborative Research Group, & et al. (1999). Use of the brain parenchymal fraction to measure whole brain atrophy in relapsing-remitting MS. Neurology, 53(8), 1698–1698.

    Article  CAS  PubMed  Google Scholar 

  • Salem, M., Valverde, S., Cabezas, M., Pareto, D., Oliver, A., Salvi, J., Rovira À., & Lladó, X. (2019). Multiple Sclerosis Lesion Synthesis in MRI using an encoder-decoder U-NET. IEEE Access.

  • Sharma, S., Rousseau, F., Heitz, F., Rumbach, L., & Armspach, J.P. (2013). On the estimation and correction of bias in local atrophy estimations using example atrophy simulations. Computerized Medical Imaging and Graphics, 37(7–8), 538–551.

    Article  PubMed  Google Scholar 

  • Shin, H.C., Tenenholtz, N.A., Rogers, J.K., Schwarz, C.G., Senjem, M.L., Gunter, J.L., Andriole, K., & Michalski, M. (2018). Medical image synthesis for data augmentation and anonymization using generative adversarial networks. In International workshop on simulation and synthesis in medical imaging (pp. 1–11): Springer.

  • Smith, S.M., Zhang, Y., Jenkinson, M., Chen, J., Matthews, P., Federico, A., & De Stefano, N. (2002). Accurate, robust, and automated longitudinal and cross-sectional brain change analysis. NeuroImage, 17(1), 479–489.

    Article  PubMed  Google Scholar 

  • Springenberg, J.T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2015). Striving for simplicity: the all convolutional net. In ICLR (workshop track) (pp. 1–14).

  • Steenwijk, M.D., Geurts, J.J.G., Daams, M., Tijms, B.M., Wink, A.M., Balk, L.J., Tewarie, P.K., Uitdehaag, B.M.J., Barkhof, F., Vrenken, H., & et al. (2016). Cortical atrophy patterns in multiple sclerosis are non-random and clinically relevant. Brain, 139(1), 115–126.

    Article  PubMed  Google Scholar 

  • Storelli, L., Rocca, M.A., Pagani, E., Van Hecke, W., Horsfield, M.A., De Stefano, N., Rovira, A., Sastre-Garriga, J., Palace, J., Sima, D., & et al. (2018). Measurement of whole-brain and gray matter atrophy in multiple sclerosis: assessment with MR imaging. Radiology, 288(2), 554–564.

    Article  PubMed  Google Scholar 

  • Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A., & et al. (2015). Going deeper with convolutions. In IEEE conference on computer vision and pattern recognition.

  • Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A.A. (2017). Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI (Vol. 4 p. 12).

  • Trottier, L., Gigu, P., Chaib-draa, B., & et al. (2017). Parametric exponential linear unit for deep convolutional neural networks. In 16th IEEE international conference on machine learning and applications (pp. 207–214).

  • van Erp, T.G., Hibar, D.P., Rasmussen, J.M., Glahn, D.C., Pearlson, G.D., Andreassen, O.A., Agartz, I., Westlye, L.T., Haukvik, U.K., Dale, A.M., & et al. (2016). Subcortical brain volume abnormalities in 2028 individuals with schizophrenia and 2540 healthy controls via the ENIGMA consortium. Molecular Psychiatry, 21(4), 547.

    Article  PubMed  Google Scholar 

  • Wang, Z., Bovik, A.C., Sheikh, H.R., & Simoncelli, E.P. (2004). Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612.

    Article  PubMed  Google Scholar 

  • Wei, W., Poirion, E., Bodini, B., Durrleman, S., Colliot, O., Stankoff, B., & Ayache, N. (2018). FLAIR MR image synthesis by using 3D fully convolutional networks for multiple sclerosis. In ISMRM-ESMRMB 2018-joint annual meeting (pp. 1–6).

  • Zhang, Y., Brady, M., & Smith, S. (2001). Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithm. IEEE Transactions on Medical Imaging, 20 (1), 45–57.

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

Data used in the preparation of this article were [in part] obtained from the OASIS dataset: Longitudinal: Principal Investigators: D. Marcus, R, Buckner, J. Csernansky, J. Morris; P50 AG05681, P01 AG03991, P01 AG026276, R01 AG021910, P20 MH071616, U24 RR021382. Data collection and sharing for the ADNI project was funded by the Alzheimer’s Disease Neuroimaging Initiative (ADNI) (National Institutes of Health Grant U01 AG024904) and DOD ADNI (Department of Defense award number W81XWH-12-2-0012). ADNI is funded by the National Institute on Aging, the National Institute of Biomedical Imaging and Bioengineering, and through generous contributions from the following: AbbVie, Alzheimer’s Association; Alzheimer’s Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company; CereSpir, Inc.; Cogstate; Eisai Inc.; Elan Pharmaceuticals, Inc.; Eli Lilly and Company; EuroImmun; F. Hoffmann-La Roche Ltd and its affiliated company Genentech, Inc.; Fujirebio; GE Healthcare; IXICO Ltd.; Janssen Alzheimer Immunotherapy Research & Development, LLC.; Johnson & Johnson Pharmaceutical Research & Development LLC.; Lumosity; Lundbeck; Merck & Co., Inc.; Meso Scale Diagnostics, LLC.; NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical Company; and Transition Therapeutics. The Canadian Institutes of Health Research is providing funds to support ADNI clinical sites in Canada. Private sector contributions are facilitated by the Foundation for the National Institutes of Health (www.fnih.org). The grantee organisation is the Northern California Institute for Research and Education, and the study is coordinated by the Alzheimer’s Therapeutic Research Institute at the University of Southern California. ADNI data are disseminated by the Laboratory for Neuro Imaging at the University of Southern California.

Author information

Authors and Affiliations

Authors

Consortia

Corresponding author

Correspondence to Jose Bernal.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

JB and KK held FI-DGR2017 grants from the Catalan Government with reference numbers 2017FI B00476 and 2017FI B00372, respectively. MC holds a Juan de la Cierva - Incorporación grant from the Spanish Government with reference number IJCI-2016-29240. This work has been partially supported by Retos de Investigació TIN2015-73563-JIN and DPI2017-86696-R from the Ministerio de Ciencia y Tecnología. Data used in the preparation of this article were [in part] obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu). As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report. A complete listing of ADNI investigators can be found at http://adni.loni.usc.edu/wp-content/uploads/how_to_apply/ADNI_Acknowledgement_List.pdf.

Electronic supplementary material

Below is the link to the electronic supplementary material.

(PDF 101 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bernal, J., Valverde, S., Kushibar, K. et al. Generating Longitudinal Atrophy Evaluation Datasets on Brain Magnetic Resonance Images Using Convolutional Neural Networks and Segmentation Priors. Neuroinform 19, 477–492 (2021). https://doi.org/10.1007/s12021-020-09499-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12021-020-09499-z

Keywords

Navigation