Skip to main content

Advertisement

SpringerLink
Log in

CaliPro: A Calibration Protocol That Utilizes Parameter Density Estimation to Explore Parameter Space and Calibrate Complex Biological Models

  • Original Article
  • Published:
Cellular and Molecular Bioengineering Aims and scope Submit manuscript

Abstract

Introduction

Mathematical and computational modeling have a long history of uncovering mechanisms and making predictions for biological systems. However, to create a model that can provide relevant quantitative predictions, models must first be calibrated by recapitulating existing biological datasets from that system. Current calibration approaches may not be appropriate for complex biological models because: 1) many attempt to recapitulate only a single aspect of the experimental data (such as a median trend) or 2) Bayesian techniques require specification of parameter priors and likelihoods to experimental data that cannot always be confidently assigned. A new calibration protocol is needed to calibrate complex models when current approaches fall short.

Methods

Herein, we develop CaliPro, an iterative, model-agnostic calibration protocol that utilizes parameter density estimation to refine parameter space and calibrate to temporal biological datasets. An important aspect of CaliPro is the user-defined pass set definition, which specifies how the model might successfully recapitulate experimental data. We define the appropriate settings to use CaliPro.

Results

We illustrate the usefulness of CaliPro through four examples including predator-prey, infectious disease transmission, and immune response models. We show that CaliPro works well for both deterministic, continuous model structures as well as stochastic, discrete models and illustrate that CaliPro can work across diverse calibration goals.

Conclusions

We present CaliPro, a new method for calibrating complex biological models to a range of experimental outcomes. In addition to expediting calibration, CaliPro may be useful in already calibrated parameter spaces to target and isolate specific model behavior for further analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7

Similar content being viewed by others

Abbreviations

ODE:

Ordinary differential equation

LHS:

Latin hypercube sampling

HDR:

Highest density region

ADS:

Alternative density subtraction

SIR:

Sample importance resampling

TB:

Tuberculosis

References

  1. Ades, A. E., et al. Bayesian methods for evidence synthesis in cost-effectiveness analysis. Pharmacoeconomics 24:1–19, 2006.

    Google Scholar 

  2. An, G. The crisis of reproducibility, the denominator problem and the scientific role of multi-scale modeling. Bull. Math. Biol. 80:3071–3080, 2018.

    Google Scholar 

  3. Azhar, N., and Y. Vodovotz. Innate immunity in disease: insights from mathematical modeling and analysis. Adv. Exp. Med. Biol. 844:227–243, 2014.

    Google Scholar 

  4. Beaumont, M. A., W. Zhang, and D. J. Balding. Approximate Bayesian computation in population genetics. Genetics 162:2025–2035, 2002.

    Google Scholar 

  5. Blum, C., and A. Roli. Metaheuristics in combinatorial optimization: overview and conceptual comparison. ACM Comput. Surv. 35:268–308, 2003.

    Google Scholar 

  6. Bohachevsky, I. O., M. E. Johnson, and M. L. Stein. Generalized simulated annealing for function optimization. Technometrics 28:209–217, 1986.

    MATH  Google Scholar 

  7. Bottou, L. Large-scale machine learning with stochastic gradient descent. Proceedings of COMPSTAT’2010, 2010, pp. 177–186.

  8. Brännmark, C., et al. Insulin signaling in type 2 diabetes: experimental and modeling analyses reveal mechanisms of insulin resistance in human adipocytes. J. Biol. Chem. 288:9867–9880, 2013.

    Google Scholar 

  9. Brewka, G. Artificial Intelligence—A Modern Approach by Stuart Russell and Peter Norvig: Series in Artificial Intelligence. Englewood Cliffs, NJ: Prentice Hall, 1996.

    Google Scholar 

  10. Cadena, A. M., S. M. Fortune, and J. L. Flynn. Heterogeneity in tuberculosis. Nat. Rev. Immunol. 17:691–702, 2017.

    Google Scholar 

  11. Castiglione, F., F. Pappalardo, C. Bianca, G. Russo, and S. Motta. Modeling biology spanning different scales: an open challenge. Biomed. Res. Int. 2014. https://doi.org/10.1155/2014/902545.

    Article  Google Scholar 

  12. Cedersund, G., and P. Strålfors. Putting the pieces together in diabetes research: towards a hierarchical model of whole-body glucose homeostasis. Eur. J. Pharm. Sci. 36:91–104, 2009.

    Google Scholar 

  13. Cilfone, N. A., C. R. Perry, D. E. Kirschner, and J. J. Linderman. Multi-scale modeling predicts a balance of tumor necrosis factor-α and interleukin-10 controls the granuloma environment during mycobacterium tuberculosis infection. PLoS ONE 2013. https://doi.org/10.1371/journal.pone.0068680.

    Article  Google Scholar 

  14. Cilfone, N. A., et al. Computational modeling predicts IL-10 control of lesion sterilization by balancing early host immunity-mediated antimicrobial responses with caseation during mycobacterium tuberculosis infection. J. Immunol. 194:664–677, 2015.

    Google Scholar 

  15. Cockrell, C., and G. An. Sepsis reconsidered: Identifying novel metrics for behavioral landscape characterization with a high-performance computing implementation of an agent-based model. J. Theor. Biol. 430:157–168, 2017.

    Google Scholar 

  16. Cornuet, J. M., J. M. Marin, A. Mira, and C. P. Robert. Adaptive multiple importance sampling. Scand. J. Stat. 39:798–812, 2012.

    MathSciNet  MATH  Google Scholar 

  17. Cowles, M. K., and B. P. Carlin. Markov Chain Monte Carlo convergence diagnostics: a comparative review. J. Am. Stat. Assoc. 91:883, 1996.

    MathSciNet  MATH  Google Scholar 

  18. Deng, X., and Y. Nakamura. Cancer precision medicine: from cancer screening to drug selection and personalized immunotherapy. Trends Pharmacol. Sci. 38:15–24, 2017.

    Google Scholar 

  19. Eisenberg, M. C., and H. V. Jain. A confidence building exercise in data and identifiability: modeling cancer chemotherapy as a case study. J. Theor. Biol. 431:63–78, 2017.

    MATH  Google Scholar 

  20. Fallahi-Sichani, M., M. El-Kebir, S. Marino, D.E. Kirschner, and J.J. Linderman. Multiscale computational modeling reveals a critical role for TNF-receptor 1 dynamics in tuberculosis granuloma formation. J. Immunol. 186:3472–3483, 2011. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3127549&tool=pmcentrez&rendertype=abstract.

  21. Farah, M., P. Birrell, S. Conti, and D. De Angelis. Bayesian emulation and calibration of a dynamic epidemic model for A/H1N1 influenza. J. Am. Stat. Assoc. 109:1398–1411, 2014.

    MathSciNet  Google Scholar 

  22. Friedman, A. A., A. Letai, D. E. Fisher, and K. T. Flaherty. Precision medicine for cancer with next-generation functional diagnostics. Nat. Rev. Cancer. 15:747–756, 2015.

    Google Scholar 

  23. Gábor, A., and J. R. Banga. Robust and efficient parameter estimation in dynamic models of biological systems. BMC Syst. Biol. 9:74, 2015.

    Google Scholar 

  24. Guzzetta, G., and D. Kirschner. The roles of immune memory and aging in protective immunity and endogenous reactivation of tuberculosis. PLoS ONE 2013. https://doi.org/10.1371/journal.pone.0060425.

    Article  Google Scholar 

  25. Hogue, T. S., S. Sorooshian, H. Gupta, A. Holz, and D. Braatz. A multistep automatic calibration scheme for river forecasting models. J. Hydrometeorol. 1:524–542, 2000.

    Google Scholar 

  26. Holland, J.H. Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence. Ann Arbor: Univ. Michigan Press, 1975. Available from: http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid=8929.

  27. Hyndman, R. J. Computing and graphing highest density regions. Am. Stat. 50:120–126, 1996.

    Google Scholar 

  28. Kitano, H. Systems biology: a brief overview. Science 295:1662–1664, 2002.

    Google Scholar 

  29. Kuepfer, L., R. Kerb, and A. M. Henney. Clinical translation in the virtual liver network. CPT Pharmacometrics Syst. Pharmacol. 3:e127, 2014.

    Google Scholar 

  30. Lin, P. L., and J. L. Flynn. Understanding latent tuberculosis: a moving target. J. Immunol. 185:15–22, 2010.

    Google Scholar 

  31. Lin, P. L., et al. Quantitative comparison of active and latent tuberculosis in the cynomolgus macaque model. Infect. Immun. 77:4631–4642, 2009.

    Google Scholar 

  32. Lin, P. L., et al. Sterilization of granulomas is common in active and latent tuberculosis despite within-host variability in bacterial killing. Nat. Med. 20:75–79, 2014.

    Google Scholar 

  33. Lunn, D. J., A. Thomas, N. Best, and D. Spiegelhalter. WinBUGS—a Bayesian modelling framework: concepts, structure, and extensibility. Stat. Comput. 10:325–337, 2000.

    Google Scholar 

  34. Marino, S., I. B. Hogue, C. J. Ray, and D. E. Kirschner. A methodology for performing global uncertainty and sensitivity analysis in systems biology. J. Theor. Biol. 254:178–196, 2008.

    MathSciNet  MATH  Google Scholar 

  35. Marino, S., J. J. Linderman, and D. E. Kirschner. A multifaceted approach to modeling the immune response in tuberculosis. Wiley Interdiscip. Rev. Syst. Biol. Med. 3:479–489, 2011.

    Google Scholar 

  36. Marino, S., and D. Kirschner. A multi-compartment hybrid computational model predicts key roles for dendritic cells in tuberculosis infection. Computation 4:39, 2016. Available from: http://www.mdpi.com/2079-3197/4/4/39.

  37. Marino, S., et al. Computational and empirical studies predict mycobacterium tuberculosis-specific T cells as a biomarker for infection outcome. PLoS Comput. Biol. 2016. https://doi.org/10.1371/journal.pcbi.1004804.

    Article  Google Scholar 

  38. Menzies, N. A., D. I. Soeteman, A. Pandya, and J. J. Kim. Bayesian methods for calibrating health policy models: a tutorial. Pharmacoeconomics 35:613–624, 2017.

    Google Scholar 

  39. Nyman, E., et al. A hierarchical whole-body modeling approach elucidates the link between in vitro insulin signaling and in vivo glucose homeostasis. J. Biol. Chem. 286:26028–26041, 2011.

    Google Scholar 

  40. Palsson, S., et al. The development of a fully-integrated immune response model (FIRM) simulator of the immune response through integration of multiple subset models. BMC Syst. Biol. 2013. https://doi.org/10.1186/1752-0509-7-95.

    Article  Google Scholar 

  41. Pienaar, E., et al. Comparing efficacies of moxifloxacin, levofloxacin and gatifloxacin in tuberculosis granulomas using a multi-scale systems pharmacology approach. PLoS Comput. Biol. 13:e1005650, 2017.

    Google Scholar 

  42. Qu, Z., A. Garfinkel, J. N. Weiss, and M. Nivala. Multi-scale modeling in biology: how to bridge the gaps between scales? Prog. Biophys. Mol. Biol. 107:21–31, 2011.

    Google Scholar 

  43. Raftery, A. E., and L. Bao. Estimating and projecting trends in HIV/AIDS generalized epidemics using incremental mixture importance sampling. Biometrics 66:1162–1173, 2010.

    MathSciNet  MATH  Google Scholar 

  44. Rajaona, H., et al. An adaptive Bayesian inference algorithm to estimate the parameters of a hazardous atmospheric release. Atmos. Environ. 122:748–762, 2015.

    Google Scholar 

  45. Read, M. N., K. Alden, J. Timmis, and P. S. Andrews. Strategies for calibrating models of biology. Brief. Bioinform. 2018. https://doi.org/10.1093/bib/bby092.

    Article  Google Scholar 

  46. Regev, A., et al. The human cell atlas. Elife 2017. https://doi.org/10.7554/eLife.27041.

    Article  Google Scholar 

  47. Rikard, S. M., et al. Multiscale coupling of an agent-based model of tissue fibrosis and a logic-based model of intracellular signaling. Front. Physiol. 2019. https://doi.org/10.3389/fphys.2019.01481.

    Article  Google Scholar 

  48. Rubin, D. B. Using the SIR algorithm to simulate posterior distributions. In: Bayesian Statistics, edited by J. M. Bernardo, D. V. Lindley, M. H. DeGroot, and A. F. M. Smith. New York: Oxford University Press, 1988, pp. 395–402.

    Google Scholar 

  49. Rutter, C. M., D. L. Miglioretti, and J. E. Savarino. Bayesian calibration of microsimulation models. J. Am. Stat. Assoc. 104:1338–1350, 2009.

    MathSciNet  MATH  Google Scholar 

  50. Santoni, D., M. Pedicini, and F. Castiglione. Implementation of a regulatory gene network to simulate the TH1/2 differentiation in an agent-based model of hypersensitivity reactions. Bioinformatics 24:1374–1380, 2008.

    Google Scholar 

  51. Schliess, F., et al. Integrated metabolic spatial-temporal model for the prediction of ammonia detoxification during liver damage and regeneration. Hepatology 60:2040–2051, 2014.

    Google Scholar 

  52. Schwen, L. O., et al. Representative sinusoids for hepatic four-scale pharmacokinetics simulations. PLoS ONE 2015. https://doi.org/10.1371/journal.pone.0133653.

    Article  Google Scholar 

  53. Segovia-Juarez, J. L., S. Ganguli, and D. Kirschner. Identifying control mechanisms of granuloma formation during M. tuberculosis infection using an agent-based model. J. Theor. Biol. 231:357–376, 2004.

    MathSciNet  MATH  Google Scholar 

  54. Spinosa, P. C., et al. Short-term cellular memory tunes the signaling responses of the chemokine receptor CXCR4. Sci. Signal. 2019. https://doi.org/10.1126/scisignal.aaw4204.

    Article  Google Scholar 

  55. Steele, R. J., A. E. Raftery, and M. J. Emond. Computing normalizing constants for finite mixture models via incremental mixture importance sampling (IMIS). J. Comput. Graph. Stat. 15:712–734, 2006.

    MathSciNet  Google Scholar 

  56. Sud, D., C. Bigbee, J. L. Flynn, and D. E. Kirschner. Contribution of CD8+ T cells to control of mycobacterium tuberculosis infection. J. Immunol. 176:4296–4314, 2014.

    Google Scholar 

  57. Sunnåker, M., A. G. Busetto, E. Numminen, J. Corander, M. Foll, and C. Dessimoz. Approximate Bayesian computation. PLoS Comput. Biol. 2013. https://doi.org/10.1371/journal.pcbi.1002803.

    Article  MathSciNet  Google Scholar 

  58. Toro, M., and J. Aracil. Chaotic behavior in predator–prey–food system dynamics models. Proc. 1986 Int. Conf. Syst. Dyn. Soc. Syst. Dyn. Move., 1986, p. 353.

  59. Wang, Q. J. The genetic algorithm and its application to calibrating conceptual rainfall-runoff models. Water Resour. Res. 27:2467–2471, 1991.

    Google Scholar 

  60. Warsinske, H. C., E. Pienaar, J. J. Linderman, J. T. Mattila, and D. E. Kirschner. Deletion of TGF-β1 increases bacterial clearance by cytotoxic t cells in a tuberculosis granuloma model. Front. Immunol. 2017. https://doi.org/10.3389/fimmu.2017.01843.

    Article  Google Scholar 

  61. Wessler, T., et al. A computational model tracks whole-lung Mycobacterium tuberculosis infection and predicts factors that inhibit dissemination. PLOS Comput. Biol. 2020. https://doi.org/10.1371/journal.pcbi.1007280.

    Article  Google Scholar 

  62. Whyte, S., C. Walsh, and J. Chilcott. Bayesian calibration of a natural history model with application to a population model for colorectal cancer. Med. Decis. Mak. 2011. https://doi.org/10.1177/0272989X10384738.

    Article  Google Scholar 

  63. Wigginton, J. E., and D. Kirschner. A model to predict cell-mediated immune regulatory mechanisms during human infection with Mycobacterium tuberculosis. J. Immunol. 166:1951–1967, 2001.

    Google Scholar 

  64. Wong, E. A., et al. Low levels of T cell exhaustion in tuberculous lung granulomas. Infect. Immun. 2018. https://doi.org/10.1128/IAI.00426-18.

    Article  Google Scholar 

Download references

Acknowledgements

This research was supported by NIH Grants R01AI123093 (DEK) and U01 HL131072 awarded to DEK and JJL. Simulations also use resources of the National Energy Research Scientific Computing Center, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. ACI-1053575 and the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation Grant MCB140228

Conflict of interest

LRJ, DEK, and JJL declare that they have no conflicts of interest.

Human Studies

No human studies were carried out by the authors for this article.

Animal Studies

No animal studies were carried out by the authors for this article.

Author information

Authors and Affiliations

  1. Department of Chemical Engineering, University of Michigan, G045W NCRC B28, 2800 Plymouth Rd, Ann Arbor, MI, 48109-2136, USA

    Louis R. Joslyn & Jennifer J. Linderman

  2. Department of Microbiology and Immunology, University of Michigan Medical School, 1150 W Medical Center Drive, 5641 Medical Science II, Ann Arbor, MI, 48109-5620, USA

    Louis R. Joslyn & Denise E. Kirschner

Authors
  1. Louis R. Joslyn
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Denise E. Kirschner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jennifer J. Linderman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Denise E. Kirschner or Jennifer J. Linderman.

Additional information

Associate Editor Michael R. King oversaw the review of this article.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 11 kb)

Supplementary material 2 (PDF 137 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Joslyn, L.R., Kirschner, D.E. & Linderman, J.J. CaliPro: A Calibration Protocol That Utilizes Parameter Density Estimation to Explore Parameter Space and Calibrate Complex Biological Models. Cel. Mol. Bioeng. 14, 31–47 (2021). https://doi.org/10.1007/s12195-020-00650-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12195-020-00650-z

Keywords

Search

Navigation