Skip to main content
Log in

Bayesian system ID: optimal management of parameter, model, and measurement uncertainty

  • Original paper
  • Published:
Nonlinear Dynamics Aims and scope Submit manuscript

Abstract

System identification of dynamical systems is often posed as a least squares minimization problem. The aim of these optimization problems is typically to learn either propagators or the underlying vector fields from trajectories of data. In this paper, we study a first principles derivation of appropriate objective formulations for system identification based on probabilistic principles. We compare the resulting inference objective to those used by emerging data-driven methods based on dynamic mode decomposition (DMD) and system identification of nonlinear dynamics (SINDy). We show that these and related least squares formulations are specific cases of a more general objective function. We also show that the more general objective function yields more robust and reliable recovery in the presence of sparse data and noisy measurements. We attribute this success to an explicit accounting of imperfect model forms, parameter uncertainty, and measurement uncertainty. We study the computational complexity of an approximate marginal Markov Chain Monte Carlo method to solve the resulting inference problem and numerically compare our results on a number of canonical systems: linear pendulum, nonlinear pendulum, the Van der Pol oscillator, the Lorenz system, and a reaction–diffusion system. The results of these comparisons show that in cases where DMD and SINDy excel, the Bayesian approach performs equally well, and in cases where DMD and SINDy fail to produce reasonable results, the Bayesian approach remains robust and can still deliver reliable results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Notes

  1. For this linear problem, it is more appropriate to consider frequency-domain system ID, which would not encounter the problems described here. However, these types of time-domain system ID procedures using least squares-based regression/machine learning approaches are increasingly being used for complex nonlinear systems [36,37,38], and we seek to show that they can be limited in an extremely simple setting.

  2. We only consider the naive matrix-multiplication scheme, not the asymptotically more optimal approaches such as Strassen’s algorithm.

References

  1. Glahn, H.R., Lowry, D.A.: The use of model output statistics (MOS) in objective weather forecasting. J. Appl. Meteorol. 11(8), 1203 (1972)

    Google Scholar 

  2. Chitsazan, M.A., Fadali, M.S., Trzynadlowski, A.M.: Wind speed and wind direction forecasting using echo state network with nonlinear functions. Renew. Energy 131, 879 (2019)

    Google Scholar 

  3. Scher, S.: Toward data-driven weather and climate forecasting: approximating a simple general circulation model with deep learning. Geophys. Res. Lett. 45(22), 12 (2018)

    Google Scholar 

  4. Kevrekidis, I.G., Gear, C.W., Hyman, J.M., Kevrekidid, P.G., Runborg, O., Theodoropoulos, C., et al.: Equation-free, coarse-grained multiscale computation: enabling mocroscopic simulators to perform system-level analysis. Commun. Math. Sci. 1(4), 715 (2003)

    MathSciNet  MATH  Google Scholar 

  5. Raissi, M.: Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Mach. Learn. Res. 19(1), 932 (2018)

    Google Scholar 

  6. Takeishi, N., Kawahara, Y., Yairi, T.: Learning Koopman invariant subspaces for dynamic mode decomposition. In Advances in Neural Information Processing Systems, pp. 1130–1140 (2017)

  7. Liu, Y.J., Li, J., Tong, S., Chen, C.P.: Neural network control-based adaptive learning design for nonlinear systems with full-state constraints. IEEE Trans. Neural Netw. Learn. Syst. 27(7), 1562 (2016)

    MathSciNet  Google Scholar 

  8. Cui, R., Yang, C., Li, Y., Sharma, S.: Adaptive neural network control of AUVs with control input nonlinearities using reinforcement learning. IEEE Trans. Syst. Man Cybern. Syst. 47(6), 1019 (2017)

    Google Scholar 

  9. Sun, K., Jianbin, Q., Karimi, H.R., Fu, Y.: Event-triggered robust fuzzy adaptive finite-time control of nonlinear systems with prescribed performance. IEEE Trans. Fuzzy Syst. (2020). https://doi.org/10.1109/TFUZZ.2020.2979129

    Article  Google Scholar 

  10. Polycarpou, M.M., Ioannou, P.A.: A robust adaptive nonlinear control design: a robust adaptive nonlinear control design. In 1993 American Control Conference, pp. 1365–1369. IEEE (1993)

  11. Ott, E., Hunt, B.R., Szunyogh, I., Zimin, A.V., Kostelich, E.J., Corazza, M., Kalnay, E., Patil, D., Yorke, J.A.: A local ensemble Kalman filter for atmospheric data assimilation. Tellus A Dyn. Meteorol. Oceanogr. 56(5), 415 (2004)

    Google Scholar 

  12. Hunt, B., Kalnay, E., Kostelich, E., Ott, E., Patil, D., Sauer, T., Szunyogh, I., Yorke, J., Zimin, A.: Four-dimensional ensemble Kalman filtering. Tellus A 56(4), 273 (2004)

    Google Scholar 

  13. Sirovich, L.: Turbulence and the dynamics of coherent structures. I. Coherent structures. Q. Appl. Math. 45(3), 561 (1987)

    MathSciNet  MATH  Google Scholar 

  14. Lumley, J.L.: Stochastic Tools in Turbulence. Courier Corporation, North Chelmsford (2007)

    MATH  Google Scholar 

  15. Schmid, P.J.: Dynamic mode decomposition of numerical and experimental data. J. Fluid Mech. 656, 5–28 (2010)

    MathSciNet  MATH  Google Scholar 

  16. Leffens, E., Markley, F., Shuster, M.: Kalman filtering for spacecraft attitude estimation. J. Guid. Control Dyn. 5(5), 417 (1982)

    Google Scholar 

  17. Slotine, J.J.E., Li, W.: On the adaptive control of robot manipulators. Int. J. Robot. Res. 6(3), 49 (1987)

    Google Scholar 

  18. Craig, J.J., Hsu, P., Sastry, S.S.: Adaptive control of mechanical manipulators. Int. J. Robot. Res. 6(2), 16 (1987)

    Google Scholar 

  19. Li, K., Kou, J., Zhang, W.: Deep neural network for unsteady aerodynamic and aeroelastic modeling across multiple Mach numbers. Nonlinear Dyn. 96(3), 2157 (2019)

    Google Scholar 

  20. De Paula, N., Marques, F.: Multi-variable Volterra kernels identification using time-delay neural networks: application to unsteady aerodynamic loading. Nonlinear Dyn. 97(1), 767 (2019)

    MATH  Google Scholar 

  21. Li, W., Laima, S., Jin, X., Yuan, W., Li, H.: A novel long short-term memory neural-network-based self-excited force model of limit cycle oscillations of nonlinear flutter for various aerodynamic configurations. Nonlinear Dyn. 100, 2071–2087 (2020)

    Google Scholar 

  22. Särkkä, S.: Bayesian Filtering and Smoothing. Institute of Mathematical Statistics Textbooks. Cambridge University Press, Cambridge (2013)

    MATH  Google Scholar 

  23. Law, K., Stuart, A., Zygalakis, K.: Data Assimilation. Springer, Cham (2015)

    MATH  Google Scholar 

  24. Barfoot, T.D.: State Estimation for Robotics. Cambridge University Press, Cambridge (2017)

    Google Scholar 

  25. Berger, J.O.: Statistical Decision Theory and Bayesian Analysis. Springer, New York (1985)

    MATH  Google Scholar 

  26. Brunton, S.L., Proctor, J.L., Kutz, J.N.: Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proc. Nat. Acad. Sci. 113(15), 3932 (2016)

    MathSciNet  MATH  Google Scholar 

  27. Erazo, K., Nagarajaiah, S.: An offline approach for output-only Bayesian identification of stochastic nonlinear systems using unscented Kalman filtering. J. Sound Vib. 397, 222–240 (2018)

    Google Scholar 

  28. Haario, H., Kalachev, L., Hakkarainen, J.: Generalized correlation integral vectors: a distance concept for chaotic dynamical systems. Chaos Interdiscip. J. Nonlinear Sci. 25(6), 063102 (2015)

    MathSciNet  Google Scholar 

  29. Noh, S.: Posterior inference on parameters in a nonlinear DSGE model via Gaussian-based filters. Comput. Econ. (2019). https://doi.org/10.1007/s10614-019-09944-5

    Article  Google Scholar 

  30. Drovandi, C., Everitt, R.G., Golightly, A., Prangle, D.: Ensemble MCMC: Accelerating Pseudo-Marginal MCMC for State Space Models using the Ensemble Kalman Filter. arXiv preprint arXiv:1906.02014 (2019)

  31. Khalil, M., Sarkar, A., Adhikari, S., Poirel, D.: The estimation of time-invariant parameters of noisy nonlinear oscillatory systems. J. Sound Vib. 344, 81–100 (2015)

    Google Scholar 

  32. Andrieu, C., Roberts, G.O.: The Pseudo-marginal approach for efficient Monte Carlo computations. Ann. Stat. 37(2), 697 (2009)

    MathSciNet  MATH  Google Scholar 

  33. Gelman, A.: Prior distributions for variance parameters in hierarchical models. Bayesian Anal. 1, 515–533 (2006)

    MathSciNet  MATH  Google Scholar 

  34. Hemati, M., Rowley, C., Cattafesta, L.: De-biasing the dynamic mode decomposition for applied Koopman spectral analysis. Theor. Comput. Fluid Dyn. 10, 10 (2017). https://doi.org/10.1007/s00162-017-0432-2

    Article  Google Scholar 

  35. Chartrand, R., Appl, I.S.R.N.: Numerical differentiation of noisy. Nonsmooth Data Math. (2011). https://doi.org/10.5402/2011/164564

    Article  Google Scholar 

  36. Yoshida, K., Takamatsu, H., Matsumoto, S.: Nonlinear identification of torsional driveshaft vibrations in a full-scale automotive vehicle during acceleration. Nonlinear Dyn. 86(1), 711 (2016)

    Google Scholar 

  37. Cheng, C., Peng, Z., Dong, X., Zhang, W., Meng, G.: Nonlinear system identification using Kautz basis expansion-based Volterra-PARAFAC model. Nonlinear Dyn. 94(3), 2277 (2018)

    Google Scholar 

  38. Yuan, L., Yang, Q., Zeng, C.: Chaos detection and parameter identification in fractional-order chaotic systems with delay. Nonlinear Dyn. 73(1–2), 439 (2013)

    MathSciNet  MATH  Google Scholar 

  39. Venkataraman, H.K., Seiler, P.J.: Recovering robustness in model-free reinforcement learning. In: 2019 American Control Conference (ACC), pp. 4210–4216. IEEE (2019)

  40. Peng, H., Li, L., Yang, Y., Liu, F.: Parameter estimation of dynamical systems via a chaotic ant swarm. Phys. Rev. E 81(1), 016207 (2010)

    Google Scholar 

  41. Evensen, G., Dee, D.P.,Schröter, J.: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. In: Ocean Modeling and Parameterization, pp. 373–398. Springer (1998)

  42. Wu, K., Xiu, D.: Data-driven deep learning of partial differential equations in modal space. J. Comput. Phys. 408, 109307 (2020)

    MathSciNet  Google Scholar 

  43. Constantine, P.G., Wang, Q.: Residual minimizing model interpolation for parameterized nonlinear dynamical systems. SIAM J. Sci. Comput. 34(4), A2118 (2012)

    MathSciNet  MATH  Google Scholar 

  44. Chen, T.Q., Rubanova, Y., Bettencourt, J., Duvenaud, D.K.: Neural ordinary differential equations. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 31, pp. 6571–6583. Curran Associates Inc, Red Hook (2018)

    Google Scholar 

  45. Tsoulos, I.G., Gavrilis, D., Glavas, E.: Solving differential equations with constructed neural networks. Neurocomputing 72(10–12), 2385 (2009)

    Google Scholar 

  46. Lagaris, I.E., Likas, A., Fotiadis, D.I.: Artificial neural networks for solving ordinary and partial differential equations. IEEE Trans. Neural Netw. 9(5), 987 (1998)

    Google Scholar 

  47. Proctor, J.L., Brunton, S.L., Kutz, J.N.: Dynamic mode decomposition with control, dynamic mode decomposition with control SIAM. J. Appl. Dyn. Syst. 15, 142 (2014)

    MATH  Google Scholar 

  48. Rowley, C.W., Mezić, I., Bagheri, S., Schlatter, P., Henningson, D.S.: Spectral analysis of nonlinear flows. J. Fluid Mech. 641, 115–127 (2009)

    MathSciNet  MATH  Google Scholar 

  49. Golub, G.H., Loan, C.F.V.: An analysis of the total least squares problem. SIAM J. Numer. Anal. 17(6), 883 (1980)

    MathSciNet  MATH  Google Scholar 

  50. Huffel, S.V., Vandewalle, J.: Analysis and properties of the generalized total least squares problem \(AX \approx B\) when some or all columns in \(A\) are subject to error. SIAM J. Matrix Anal. Appl. 10(3), 294 (1989)

    MathSciNet  MATH  Google Scholar 

  51. Takeishi, N., Kawahara, Y., Tabei, Y., Yairi, T.: Bayesian dynamic mode decomposition. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pp. 2814–2821 (2017)

  52. Evensen, G.: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res. Oceans 99(C5), 10143 (1994)

    Google Scholar 

  53. Arasaratnam, I., Haykin, S.: Cubature Kalman filters. IEEE Trans. Autom. Control 54(6), 1254 (2009)

    MathSciNet  MATH  Google Scholar 

  54. Julier, S.J., Uhlmann, J.K.: New extension of the Kalman filter to nonlinear systems. In: I. Kadar (ed.) Signal Processing, Sensor Fusion, and Target Recognition VI, vol. 3068. International Society for Optics and Photonics (SPIE), vol. 3068, pp. 182–193 (1997)

  55. Gordon, N.J., Salmond, D.J., Smith, A.F.: Novel approach to nonlinear/non-Gaussian Bayesian state estimation. In: IEE Proceedings F (Radar and Signal Processing), vol. 140, pp. 107–113. IET (1993)

  56. Andrieu, C., Doucet, A., Holenstein, R.: Particle Markov chain Monte Carlo methods. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 72(3), 269 (2010)

    MathSciNet  MATH  Google Scholar 

  57. Haario, H., Laine, M., Mira, A., Saksman, E.: Chaotic dynamical systems. Stat. Comput. 16, 339 (2006)

    MathSciNet  Google Scholar 

  58. Houtzager, I.: Total least squares with mixed and/or weighted disturbances. MATLAB File Exchange (2019). Retrieved 5 Dec 2019

  59. Colton, D., Kress, R.: Inverse Acoustic and Electromagnetic Scattering Theory, vol. 93. Springer, New York (2019)

    MATH  Google Scholar 

  60. Wirgin, A.: arXiv preprint arXiv:math-ph/0401050 (2004)

  61. Chen, Y., Pi, D., Wang, B.: Enhanced global flower pollination algorithm for parameter identification of chaotic and hyper-chaotic system. Nonlinear Dyn. 97(2), 1343 (2019)

    MATH  Google Scholar 

  62. Lu, Z.R., Liu, G., Liu, J., Chen, Y.M., Wang, L.: Parameter identification of nonlinear fractional-order systems by enhanced response sensitivity approach. Nonlinear Dyn. 95(2), 1495 (2019)

    MATH  Google Scholar 

  63. Narayanan, M., Narayanan, S., Padmanabhan, C.: Parametric identification of nonlinear systems using multiple trials. Nonlinear Dyn. 48(4), 341 (2007)

    MATH  Google Scholar 

  64. Marzouk, Y.M., Najm, H.N., Rahn, L.A.: Stochastic spectral methods for efficient Bayesian solution of inverse problems. J. Comput. Phys. 224(2), 560 (2007)

    MathSciNet  MATH  Google Scholar 

  65. Marzouk, Y.M., Najm, H.N.: Dimensionality reduction and polynomial chaos acceleration of Bayesian inference in inverse problems. J. Comput. Phys. 228(6), 1862 (2009)

    MathSciNet  MATH  Google Scholar 

  66. Brastein, O.M., Perera, D.W.U., Pfeifer, C., Skeie, N.O.: Parameter estimation for grey-box models of building thermal behaviour. Energy Build. 169, 58 (2018)

    Google Scholar 

  67. Dokos, S., Lovell, N.H.: Parameter estimation in cardiac ionic models. Prog. Biophys. Mol. Biol. 85(2–3), 407 (2004)

    Google Scholar 

  68. Kivman, G.A.: Sequential parameter estimation for stochastic systems. Nonlinear Processes Geophys. 10(3), 253 (2003)

    Google Scholar 

  69. Lorenz, E.N.: Deterministic nonperiodic flow. J. Atmos. Sci. 20(2), 130 (1963)

    MathSciNet  MATH  Google Scholar 

  70. Springer, S., Haario, H., Shemyakin, V., Kalachev, L., Shchepakin, D.: Robust parameter estimation of chaotic systems. Inverse Probl. Imaging 13(6), 1189 (2019)

    MathSciNet  MATH  Google Scholar 

  71. Politi, A.: Lyapunov exponent. Scholarpedia 8(3), 2722 (2013). https://doi.org/10.4249/scholarpedia.2722. Revision #137286

    Article  Google Scholar 

  72. Govorukhin, V.: Calculation Lyapunov exponents for ode. MATLAB File Exchange (2020). Retrieved 29 June 2020

  73. Wolf, A., Swift, J.B., Swinney, H.L., Vastano, J.A.: Determining Lyapunov exponents from a time series. Physica D 16(3), 285 (1985)

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This research was primarily supported by the DARPA Physics of AI Program under the grant “Physics Inspired Learning and Learning the Order and Structure of Physics,” Agreement No. HR00111890030. It was also supported in part by the DARPA Artificial Intelligence Research Associate under the grant “Artificial Intelligence Guided Multi-scale Multi-physics Framework for Discovering Complex Emergent Materials Phenomena,” Agreement No. HR00111990028.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Nicholas Galioto.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Pseudocode

Pseudocode

In this appendix, we provide the pseudocode for both the linear Kalman filter and nonlinear unscented Kalman filter algorithms. In the UKF algorithm, \(\alpha \) and \(\kappa \) are parameters that determine the spread of the sigma points around the mean, \(\beta \) is a parameter used for incorporating prior information on the distribution of x, and the notation \([\cdot ]_i\) denotes the i-th row of the matrix [22].

figure b
figure c

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Galioto, N., Gorodetsky, A.A. Bayesian system ID: optimal management of parameter, model, and measurement uncertainty. Nonlinear Dyn 102, 241–267 (2020). https://doi.org/10.1007/s11071-020-05925-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11071-020-05925-8

Keywords

Navigation