Abstract
In this paper, we show that reproducibility is a severe problem that concerns simulation models. The reproducibility problem challenges the concept of numerical solution and hence the conception of what a simulation actually does. We provide an expanded picture of simulation that makes visible those steps of simulation modeling that are numerically relevant, but often escape notice in accounts of simulation. Examining these steps and analyzing a number of pertinent examples, we argue that numerical solutions are importantly different from usual mathematical solutions. They are do not merely approximate the latter, but introduce new problems, including issues of artificiality, stability, and well-posedness. Consequently, simulation modelling can attain reproducibility only to a certain degree because it is working with numerical solutions (in a sense we specify in the paper).
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
There is a less friendly interpretation of the crisis in psychology that says results cannot be reproduced because the publications include outright fraud. No formal models of any kind can be an effective remedy then.
If reproducibility (across different groups and machines) is discerned from repeatability (same group, same machine), then the clients expect repeatability.
If adjusting parameters is automated, it can be conceived as an a-iteration.
Lenhard (2007) examines another case where a pioneer of simulation met resistance because of the artificial elements he advocated.
References
Böning, C., Beismann, J.-O., Biastoch, A., Czeschel, L., & Dengg, J. (2002). Ozeanische Aufnahme anthropogener Spurengase: Realistische Darstellung des Effektes mesoskaliger Prozesse in Zirkulationsmodellen. Presentation at DKRZ WLA-Workshop, 24(10), 2002.
Collins, H. (1985). Replication and induction in scientific practice. Chicago: Chicago University Press.
Duddeck, F. (2007) Survey on robust design and optimization for crashworthiness. In EUROMECH colloquium 482: Efficient methods for robust design and optimization. Queen Mary, University of London, London, UK.
Fillion, N. (2017). The vindication of computer simulations. In J. Lenhard, M. Carrier (eds.) Mathematics as a tool (pp. 137–55). Boston Studies in History and Philosophy of Science 327. New York: Springer.
Fillion, N., & Corless, R. M. (2014). On the epistemological analysis of modeling and computational error in the mathematical sciences. Synthese, 191, 1451–1467.
Hasse, H., & Lenhard, J. (2017). Boon and bane. On the role of adjustable parameters in simulation models. In J. Lenhard, & M. Carrier (eds.) Mathematics as a tool. Tracing new roles of mathematics in the sciences. Boston Studies in the Philosophy and History of Science (Vol. 327, pp. 93–115).
Ioannidis, J. P. (2005). Contradicted and initially stronger effects in highly cited clinical research. JAMA, 294(2), 218–228.
Kaminski, A., Resch, M., and Küster, U.: Mathematische Opazität. Über Rechtfertigung und Reproduzierbarkeit in der Computersimulation. In A. Friedrich, P. Gehring, C. Hubig, A. Kaminski, & A. Nordmann (eds.) Jahrbuch Technikphilosophie (Vol. 4, pp. 253–277). Nomos Verlag, 2018.
Lejaeghere, K., Bihlmayer, G., Björkman, T., Blaha, P., Blügel, S., Blum, V., et al. (2016). Reproducibility in density functional theory calculations of solids. Science, 351, 3000.
Lenhard, J. (2007). Computer simulation: The cooperation between experimenting and modeling. Philosophy of Science, 74, 176–194.
Lenhard, J. (2016). Computer simulation. In P. Humphreys (Ed.), Oxford handbook in the philosophy of science (pp. 717–737). New York: Oxford University Press.
Ludwig, T. (2017). Reproducibility in science, computer science and climate science. News from Computational Climate Science. Prsentation in: Leogang, 08.03.2017.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), 4716.
Parker, W. S. (2013). Computer Simulation. In S. Psillos & M. Curd (Eds.), The routledge companion to philosophy of science (2d ed., pp. 135–145). New York: Routledge.
Richardson, L. F. (1922). Weather prediction by numerical process, 1st edn. 1922, Cambridge UP; cited according to second unaltered and unabridged edition with new introduction by Sydney Chapman, 1965, Dover.
Schappals, M., Mecklenfeld, A., Kröger, L., Botan, V., Köster, A., Stephan, S., et al. (2017). Round Robin study: Molecular simulation of thermodynamic properties from models with internal degrees of freedom. Journal of Chemical Theory and Computation, 13, 4270–4280. https://doi.org/10.1021/acs.jctc.7b00489.
Solomon, M. (2011). Just a paradigm: Evidence-based medicien in epistemological context. European Journal for Philosophy of Science, 1, 451–466.
Winsberg, E. (1999). Sanctioning models: The epistemology of simulation. Science in Context, 12(2), 275–292.
Winsberg, E. (2003). Simulated experiments: Methodology for a virtual world. Philosophy of Science, 70, 105–125.
Winsberg, E. (2018). Computer simulations in science. In Edward N. Zalta (ed.) The stanford encyclopedia of philosophy (Summer 2018 Edition), forthcoming. https://plato.stanford.edu/archives/sum2018/entries/simulations-science/.
Acknowledgements
We like to thank three anonymous reviewers for useful suggestions and Nicholas Danne for his support in approximating our text to English language.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Lenhard, J., Küster, U. Reproducibility and the Concept of Numerical Solution. Minds & Machines 29, 19–36 (2019). https://doi.org/10.1007/s11023-019-09492-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11023-019-09492-9