Abstract
The predictive power of machine learning models often exceeds that of mechanistic modeling approaches. However, the interpretability of purely data-driven models, without any mechanistic basis is often complicated, and predictive power by itself can be a poor metric by which we might want to judge different methods. In this work, we focus on the relatively new modeling techniques of neural ordinary differential equations. We discuss how they relate to machine learning and mechanistic models, with the potential to narrow the gulf between these two frameworks: they constitute a class of hybrid model that integrates ideas from data-driven and dynamical systems approaches. Training neural ODEs as representations of dynamical systems data has its own specific demands, and we here propose a collocation scheme as a fast and efficient training strategy. This alleviates the need for costly ODE solvers. We illustrate the advantages that collocation approaches offer, as well as their robustness to qualitative features of a dynamical system, and the quantity and quality of observational data. We focus on systems that exemplify some of the hallmarks of complex dynamical systems encountered in systems biology, and we map out how these methods can be used in the analysis of mathematical models of cellular and physiological processes.
Acknowledgment
We gratefully acknowledge discussions with members of the Theoretical Systems Biology group at University of Melbourne, Australia, and at Imperial College London, United Kingdom, as well as with the Julia community. The information, data, or work presented herein was funded in part by ARPA-E under award numbers DE-AR0001222 and DE-AR0001211, and NSF award number IIP-1938400. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof.
Author contribution: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.
Research funding: None declared.
Conflict of interest statement: The authors declare no conflicts of interest regarding this article.
References
Álvarez et al., 2010 Álvarez, M., Luengo, D., Titsias, M., and Lawrence, N. (2010). Efficient multioutput Gaussian processes through variational inducing kernels. J. Mach. Learn. Res. 9: 25–32.Search in Google Scholar
Babtie, A.C., Kirk, P., and Stumpf, M.P.H. (2014). Topological sensitivity analysis for systems biology. Proc. Natl. Acad. Sci. U.S.A. 111: 18507–18512. https://doi.org/10.1073/pnas.1414026112.Search in Google Scholar PubMed PubMed Central
Baker, R.E., Peña, J.-M., Jayamohan, J., and Jérusalem, A. (2018). Mechanistic models versus machine learning, a fight worth fighting for the biological community?. Biol. Lett. 14: 20170660. https://doi.org/10.1098/rsbl.2017.0660.Search in Google Scholar PubMed PubMed Central
Che, Z., Purushotham, S., Cho, K., Sontag, D., and Liu, Y. (2018). Recurrent neural networks for multivariate time series with missing values. Sci. Rep. 8. https://doi.org/10.1038/s41598-018-24271-9.Search in Google Scholar PubMed PubMed Central
Chen, R.T.Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. (2019). Neural ordinary differential equations. arXiv.Search in Google Scholar
Crook, O.M., Gatto, L., and Kirk, P.D.W. (2019). Fast approximate inference for variable selection in dirichlet process mixtures, with an application to pan-cancer proteomics. Stat. Appl. Genet. Mol. Biol. 18: 20180065. https://doi.org/10.1515/sagmb-2018-0065.Search in Google Scholar PubMed
Dupont, E., Doucet, A., and Teh, Y.W. (2019). Augmented neural ODEs. arXiv.Search in Google Scholar
Durbin, J. and Koopman, S.J. (2012). Time series analysis by state space methods. Oxford statistical science series, 2nd ed. Oxford University Press, Oxford.10.1093/acprof:oso/9780199641178.001.0001Search in Google Scholar
Estakhroueieh, M., Nikravesh, S., and Gharibzadeh, S. (2014). ECG generation based on action potential using modified van der pol equation. Annu. Res. Rev. Biol. 4: 4259–4272. https://doi.org/10.9734/arrb/2014/11916.Search in Google Scholar
Gardiner, C. (2009). Stochastic methods: a handbook for the natural and social sciences. Springer, Berlin and Heidelberg.Search in Google Scholar
Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. J. Mach. Learn. Res. 9: 249–256.Search in Google Scholar
Gupta, A. and Khammash, M. (2014). Sensitivity analysis for stochastic chemical reaction networks with multiple time-scales. Electron. J. Probab. 19. https://doi.org/10.1214/ejp.v19-3246.Search in Google Scholar
Heinonen, M., Yildiz, C., Mannerström, H., Intosalmi, J., and Lähdesmäki, H. (2018). Learning unknown ODE models with Gaussian processes. arXiv.Search in Google Scholar
Innes, M. (2018). Flux: elegant machine learning with julia. J. Open Source Software 3: 602. https://doi.org/10.21105/joss.00602.Search in Google Scholar
Innes, M., Saba, E., Fischer, K., Gandhi, D., Rudilosso, M.C., Joy, N.M., Karmali, T., Pal, A., and Shah, V. (2018). Fashionable modelling with flux. arXiv.Search in Google Scholar
Jia, J. and Benson, A.R. (2019). Neural jump stochastic differential equations. arXiv.Search in Google Scholar
Jost, J. (2005). Dynamical systems examples of complex behaviour. Springer, Berlin and Heidelberg.Search in Google Scholar
Kersting, H., Krämer, N., Schiegg, M., Daniel, C., Tiemann, M., and Hennig, P. (2020). Differentiable likelihoods for fast inversion of ’likelihood-free’ dynamical systems. arXiv.Search in Google Scholar
Kirk, P., Thorne, T., and Stumpf, M.P. (2013). Model selection in systems and synthetic biology. Curr. Opin. Biotechnol. 24: 767–774. https://doi.org/10.1016/j.copbio.2013.03.012.Search in Google Scholar PubMed
Kirk, P.D.W., Babtie, A.C., and Stumpf, M.P.H. (2015). Systems biology (un)certainties. Science 350: 386–388. https://doi.org/10.1126/science.aac9505.Search in Google Scholar PubMed
Lakatos, E. and Stumpf, M.P.H. (2017). Control mechanisms for stochastic biochemical systems via computation of reachable sets. R. Soc. Open Sci. 4: 160790. https://doi.org/10.1098/rsos.160790.Search in Google Scholar PubMed PubMed Central
LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learning. Nature 521: 436–444. https://doi.org/10.1038/nature14539.Search in Google Scholar PubMed
Leon, M., Woods, M.L., Fedorec, A.J.H., and Barnes, C.P. (2016). A computational method for the investigation of multistable systems and its application to genetic switches. BMC Syst. Biol. 10. https://doi.org/10.1186/s12918-016-0375-z.Search in Google Scholar PubMed PubMed Central
Liang, H. and Wu, H. (2008). Parameter estimation for differential equation models using a framework of measurement error in regression models. J. Am. Stat. Assoc. 103: 1570–1583. https://doi.org/10.1198/016214508000000797.Search in Google Scholar PubMed PubMed Central
Liepe, J., Filippi, S., Komorowski, M., and Stumpf, M.P.H. (2013). Maximizing the information content of experiments in systems biology. PLoS Comput. Biol. 9: e1002888. https://doi.org/10.1371/journal.pcbi.1002888.Search in Google Scholar PubMed PubMed Central
Liepe, J., Kirk, P., Filippi, S., Toni, T., Barnes, C.P., and Stumpf, M.P.H. (2014). A framework for parameter estimation and model selection from experimental data in systems biology using approximate bayesian computation. Nat. Protoc. 9: 439–456. https://doi.org/10.1038/nprot.2014.025.Search in Google Scholar PubMed PubMed Central
Liu, X., Xiao, T., Si, S., Cao, Q., Kumar, S., and Hsieh, C.-J. (2019). Neural SDE: stabilizing neural ODE networks with stochastic noise. arXiv.Search in Google Scholar
Milias-Argeitis, A., Summers, S., Stewart-Ornstein, J., Zuleta, I., Pincus, D., El-Samad, H., Khammash, M., and Lygeros, J. (2011). In silico feedback for in vivo regulation of a gene expression circuit. Nat. Biotechnol. 29: 1114–1116. https://doi.org/10.1038/nbt.2018.Search in Google Scholar PubMed PubMed Central
Murphy, K.P. (2012). Machine learning: a probabilistic perspective. MIT Press, Cambridge, Massachusetts and London, England.Search in Google Scholar
Rackauckas, C. and Nie, Q. (2017). DifferentialEquations.jl – a performant and feature-rich ecosystem for solving differential equations in julia. J. Open Res. Software 5. https://doi.org/10.5334/jors.151.Search in Google Scholar
Rackauckas, C., Innes, M., Ma, Y., Bettencourt, J., White, L., and Dixit, V. (2019). DiffEqFlux.jl–a julia library for neural differential equations. arXiv.Search in Google Scholar
Rackauckas, C., Ma, Y., Martensen, J., Warner, C., Zubov, K., Supekar, R., Skinner, D., and Ramadhan, A. (2020). Universal differential equations for scientific machine learning. arXiv.10.21203/rs.3.rs-55125/v1Search in Google Scholar
Rasmussen, C.E. and Williams, C.K.I. (2006). Gaussian processes for machine learning. MIT Press, Cambridge, Massachusetts and London, England.Search in Google Scholar
Roesch, E. and Stumpf, M.P.H. (2019). Parameter inference in dynamical systems with co-dimension 1 bifurcations. R. Soc. Open Sci. 6: 190747. https://doi.org/10.1098/rsos.190747.Search in Google Scholar PubMed PubMed Central
Rubanova, Y., Chen, R.T.Q., and Duvenaud, D. (2019). Latent ODEs for irregularly-sampled time series. arXiv.Search in Google Scholar
Schnoerr, D., Sanguinetti, G., and Grima, R. (2017). Approximation and inference methods for stochastic biochemical kinetics—a tutorial review. J. Phys. A: Math. Theor. 50: 093001. https://doi.org/10.1088/1751-8121/aa54d9.Search in Google Scholar
Scholes, N.S., Schnoerr, D., Isalan, M., and Stumpf, M.P.H. (2019). A comprehensive network atlas reveals that turing patterns are common but not robust. Cell Syst. 9: 243–257.e4. https://doi.org/10.1016/j.cels.2019.07.007.Search in Google Scholar
Silk, D., Barnes, C.P., Kirk, P.D.W., Kirk, P., Toni, T., Rose, A., Moon, S., Dallman, M.J., Stumpf, M.P.H., and Stumpf, M.P.H. (2011). Designing attractive models via automated identification of chaotic and oscillatory dynamical regimes. Nat. Commun. 2: 489. https://doi.org/10.1038/ncomms1496.Search in Google Scholar
Tankhilevich, E., Ish-Horowicz, J., Hameed, T., Roesch, E., Kleijn, I., Stumpf, M.P.H., and He, F. (2020). GpABC: a julia package for approximate bayesian computation with Gaussian process emulation. Bioinformatics 36: 3286–3287. https://doi.org/10.1093/bioinformatics/btaa078.Search in Google Scholar
Toni, T., Welch, D., Strelkowa, N., Ipsen, A., and Stumpf, M.P. (2008). Approximate bayesian computation scheme for parameter inference and model selection in dynamical systems. J. R. Soc. Interface 6: 187–202. https://doi.org/10.1098/rsif.2008.0172.Search in Google Scholar
Tyson, J.J., Chen, K.C., and Novák, B. (2003). Sniffers, buzzers, toggles and blinkers: dynamics of regulatory and signaling pathways in the cell. Curr. Opin. Cell Biol. 15: 221–231. https://doi.org/10.1016/s0955-0674(03)00017-6.Search in Google Scholar
Tzen, B. and Raginsky, M. (2019). Neural stochastic differential equations: deep latent Gaussian models in the diffusion limit. arXiv.Search in Google Scholar
Žurauskienė et al., 2014 Žurauskienė, J., Kirk, P., Thorne, T., and Stumpf, M.P. (2014). Bayesian non-parametric approaches to reconstructing oscillatory systems and the nyquist limit. Phys. A 407: 33–42.10.1016/j.physa.2014.03.069Search in Google Scholar
© 2021 Walter de Gruyter GmbH, Berlin/Boston
