Abstract
System identification of dynamical systems is often posed as a least squares minimization problem. The aim of these optimization problems is typically to learn either propagators or the underlying vector fields from trajectories of data. In this paper, we study a first principles derivation of appropriate objective formulations for system identification based on probabilistic principles. We compare the resulting inference objective to those used by emerging data-driven methods based on dynamic mode decomposition (DMD) and system identification of nonlinear dynamics (SINDy). We show that these and related least squares formulations are specific cases of a more general objective function. We also show that the more general objective function yields more robust and reliable recovery in the presence of sparse data and noisy measurements. We attribute this success to an explicit accounting of imperfect model forms, parameter uncertainty, and measurement uncertainty. We study the computational complexity of an approximate marginal Markov Chain Monte Carlo method to solve the resulting inference problem and numerically compare our results on a number of canonical systems: linear pendulum, nonlinear pendulum, the Van der Pol oscillator, the Lorenz system, and a reaction–diffusion system. The results of these comparisons show that in cases where DMD and SINDy excel, the Bayesian approach performs equally well, and in cases where DMD and SINDy fail to produce reasonable results, the Bayesian approach remains robust and can still deliver reliable results.
Similar content being viewed by others
Notes
For this linear problem, it is more appropriate to consider frequency-domain system ID, which would not encounter the problems described here. However, these types of time-domain system ID procedures using least squares-based regression/machine learning approaches are increasingly being used for complex nonlinear systems [36,37,38], and we seek to show that they can be limited in an extremely simple setting.
We only consider the naive matrix-multiplication scheme, not the asymptotically more optimal approaches such as Strassen’s algorithm.
References
Glahn, H.R., Lowry, D.A.: The use of model output statistics (MOS) in objective weather forecasting. J. Appl. Meteorol. 11(8), 1203 (1972)
Chitsazan, M.A., Fadali, M.S., Trzynadlowski, A.M.: Wind speed and wind direction forecasting using echo state network with nonlinear functions. Renew. Energy 131, 879 (2019)
Scher, S.: Toward data-driven weather and climate forecasting: approximating a simple general circulation model with deep learning. Geophys. Res. Lett. 45(22), 12 (2018)
Kevrekidis, I.G., Gear, C.W., Hyman, J.M., Kevrekidid, P.G., Runborg, O., Theodoropoulos, C., et al.: Equation-free, coarse-grained multiscale computation: enabling mocroscopic simulators to perform system-level analysis. Commun. Math. Sci. 1(4), 715 (2003)
Raissi, M.: Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Mach. Learn. Res. 19(1), 932 (2018)
Takeishi, N., Kawahara, Y., Yairi, T.: Learning Koopman invariant subspaces for dynamic mode decomposition. In Advances in Neural Information Processing Systems, pp. 1130–1140 (2017)
Liu, Y.J., Li, J., Tong, S., Chen, C.P.: Neural network control-based adaptive learning design for nonlinear systems with full-state constraints. IEEE Trans. Neural Netw. Learn. Syst. 27(7), 1562 (2016)
Cui, R., Yang, C., Li, Y., Sharma, S.: Adaptive neural network control of AUVs with control input nonlinearities using reinforcement learning. IEEE Trans. Syst. Man Cybern. Syst. 47(6), 1019 (2017)
Sun, K., Jianbin, Q., Karimi, H.R., Fu, Y.: Event-triggered robust fuzzy adaptive finite-time control of nonlinear systems with prescribed performance. IEEE Trans. Fuzzy Syst. (2020). https://doi.org/10.1109/TFUZZ.2020.2979129
Polycarpou, M.M., Ioannou, P.A.: A robust adaptive nonlinear control design: a robust adaptive nonlinear control design. In 1993 American Control Conference, pp. 1365–1369. IEEE (1993)
Ott, E., Hunt, B.R., Szunyogh, I., Zimin, A.V., Kostelich, E.J., Corazza, M., Kalnay, E., Patil, D., Yorke, J.A.: A local ensemble Kalman filter for atmospheric data assimilation. Tellus A Dyn. Meteorol. Oceanogr. 56(5), 415 (2004)
Hunt, B., Kalnay, E., Kostelich, E., Ott, E., Patil, D., Sauer, T., Szunyogh, I., Yorke, J., Zimin, A.: Four-dimensional ensemble Kalman filtering. Tellus A 56(4), 273 (2004)
Sirovich, L.: Turbulence and the dynamics of coherent structures. I. Coherent structures. Q. Appl. Math. 45(3), 561 (1987)
Lumley, J.L.: Stochastic Tools in Turbulence. Courier Corporation, North Chelmsford (2007)
Schmid, P.J.: Dynamic mode decomposition of numerical and experimental data. J. Fluid Mech. 656, 5–28 (2010)
Leffens, E., Markley, F., Shuster, M.: Kalman filtering for spacecraft attitude estimation. J. Guid. Control Dyn. 5(5), 417 (1982)
Slotine, J.J.E., Li, W.: On the adaptive control of robot manipulators. Int. J. Robot. Res. 6(3), 49 (1987)
Craig, J.J., Hsu, P., Sastry, S.S.: Adaptive control of mechanical manipulators. Int. J. Robot. Res. 6(2), 16 (1987)
Li, K., Kou, J., Zhang, W.: Deep neural network for unsteady aerodynamic and aeroelastic modeling across multiple Mach numbers. Nonlinear Dyn. 96(3), 2157 (2019)
De Paula, N., Marques, F.: Multi-variable Volterra kernels identification using time-delay neural networks: application to unsteady aerodynamic loading. Nonlinear Dyn. 97(1), 767 (2019)
Li, W., Laima, S., Jin, X., Yuan, W., Li, H.: A novel long short-term memory neural-network-based self-excited force model of limit cycle oscillations of nonlinear flutter for various aerodynamic configurations. Nonlinear Dyn. 100, 2071–2087 (2020)
Särkkä, S.: Bayesian Filtering and Smoothing. Institute of Mathematical Statistics Textbooks. Cambridge University Press, Cambridge (2013)
Law, K., Stuart, A., Zygalakis, K.: Data Assimilation. Springer, Cham (2015)
Barfoot, T.D.: State Estimation for Robotics. Cambridge University Press, Cambridge (2017)
Berger, J.O.: Statistical Decision Theory and Bayesian Analysis. Springer, New York (1985)
Brunton, S.L., Proctor, J.L., Kutz, J.N.: Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proc. Nat. Acad. Sci. 113(15), 3932 (2016)
Erazo, K., Nagarajaiah, S.: An offline approach for output-only Bayesian identification of stochastic nonlinear systems using unscented Kalman filtering. J. Sound Vib. 397, 222–240 (2018)
Haario, H., Kalachev, L., Hakkarainen, J.: Generalized correlation integral vectors: a distance concept for chaotic dynamical systems. Chaos Interdiscip. J. Nonlinear Sci. 25(6), 063102 (2015)
Noh, S.: Posterior inference on parameters in a nonlinear DSGE model via Gaussian-based filters. Comput. Econ. (2019). https://doi.org/10.1007/s10614-019-09944-5
Drovandi, C., Everitt, R.G., Golightly, A., Prangle, D.: Ensemble MCMC: Accelerating Pseudo-Marginal MCMC for State Space Models using the Ensemble Kalman Filter. arXiv preprint arXiv:1906.02014 (2019)
Khalil, M., Sarkar, A., Adhikari, S., Poirel, D.: The estimation of time-invariant parameters of noisy nonlinear oscillatory systems. J. Sound Vib. 344, 81–100 (2015)
Andrieu, C., Roberts, G.O.: The Pseudo-marginal approach for efficient Monte Carlo computations. Ann. Stat. 37(2), 697 (2009)
Gelman, A.: Prior distributions for variance parameters in hierarchical models. Bayesian Anal. 1, 515–533 (2006)
Hemati, M., Rowley, C., Cattafesta, L.: De-biasing the dynamic mode decomposition for applied Koopman spectral analysis. Theor. Comput. Fluid Dyn. 10, 10 (2017). https://doi.org/10.1007/s00162-017-0432-2
Chartrand, R., Appl, I.S.R.N.: Numerical differentiation of noisy. Nonsmooth Data Math. (2011). https://doi.org/10.5402/2011/164564
Yoshida, K., Takamatsu, H., Matsumoto, S.: Nonlinear identification of torsional driveshaft vibrations in a full-scale automotive vehicle during acceleration. Nonlinear Dyn. 86(1), 711 (2016)
Cheng, C., Peng, Z., Dong, X., Zhang, W., Meng, G.: Nonlinear system identification using Kautz basis expansion-based Volterra-PARAFAC model. Nonlinear Dyn. 94(3), 2277 (2018)
Yuan, L., Yang, Q., Zeng, C.: Chaos detection and parameter identification in fractional-order chaotic systems with delay. Nonlinear Dyn. 73(1–2), 439 (2013)
Venkataraman, H.K., Seiler, P.J.: Recovering robustness in model-free reinforcement learning. In: 2019 American Control Conference (ACC), pp. 4210–4216. IEEE (2019)
Peng, H., Li, L., Yang, Y., Liu, F.: Parameter estimation of dynamical systems via a chaotic ant swarm. Phys. Rev. E 81(1), 016207 (2010)
Evensen, G., Dee, D.P.,Schröter, J.: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. In: Ocean Modeling and Parameterization, pp. 373–398. Springer (1998)
Wu, K., Xiu, D.: Data-driven deep learning of partial differential equations in modal space. J. Comput. Phys. 408, 109307 (2020)
Constantine, P.G., Wang, Q.: Residual minimizing model interpolation for parameterized nonlinear dynamical systems. SIAM J. Sci. Comput. 34(4), A2118 (2012)
Chen, T.Q., Rubanova, Y., Bettencourt, J., Duvenaud, D.K.: Neural ordinary differential equations. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 31, pp. 6571–6583. Curran Associates Inc, Red Hook (2018)
Tsoulos, I.G., Gavrilis, D., Glavas, E.: Solving differential equations with constructed neural networks. Neurocomputing 72(10–12), 2385 (2009)
Lagaris, I.E., Likas, A., Fotiadis, D.I.: Artificial neural networks for solving ordinary and partial differential equations. IEEE Trans. Neural Netw. 9(5), 987 (1998)
Proctor, J.L., Brunton, S.L., Kutz, J.N.: Dynamic mode decomposition with control, dynamic mode decomposition with control SIAM. J. Appl. Dyn. Syst. 15, 142 (2014)
Rowley, C.W., Mezić, I., Bagheri, S., Schlatter, P., Henningson, D.S.: Spectral analysis of nonlinear flows. J. Fluid Mech. 641, 115–127 (2009)
Golub, G.H., Loan, C.F.V.: An analysis of the total least squares problem. SIAM J. Numer. Anal. 17(6), 883 (1980)
Huffel, S.V., Vandewalle, J.: Analysis and properties of the generalized total least squares problem \(AX \approx B\) when some or all columns in \(A\) are subject to error. SIAM J. Matrix Anal. Appl. 10(3), 294 (1989)
Takeishi, N., Kawahara, Y., Tabei, Y., Yairi, T.: Bayesian dynamic mode decomposition. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pp. 2814–2821 (2017)
Evensen, G.: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res. Oceans 99(C5), 10143 (1994)
Arasaratnam, I., Haykin, S.: Cubature Kalman filters. IEEE Trans. Autom. Control 54(6), 1254 (2009)
Julier, S.J., Uhlmann, J.K.: New extension of the Kalman filter to nonlinear systems. In: I. Kadar (ed.) Signal Processing, Sensor Fusion, and Target Recognition VI, vol. 3068. International Society for Optics and Photonics (SPIE), vol. 3068, pp. 182–193 (1997)
Gordon, N.J., Salmond, D.J., Smith, A.F.: Novel approach to nonlinear/non-Gaussian Bayesian state estimation. In: IEE Proceedings F (Radar and Signal Processing), vol. 140, pp. 107–113. IET (1993)
Andrieu, C., Doucet, A., Holenstein, R.: Particle Markov chain Monte Carlo methods. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 72(3), 269 (2010)
Haario, H., Laine, M., Mira, A., Saksman, E.: Chaotic dynamical systems. Stat. Comput. 16, 339 (2006)
Houtzager, I.: Total least squares with mixed and/or weighted disturbances. MATLAB File Exchange (2019). Retrieved 5 Dec 2019
Colton, D., Kress, R.: Inverse Acoustic and Electromagnetic Scattering Theory, vol. 93. Springer, New York (2019)
Wirgin, A.: arXiv preprint arXiv:math-ph/0401050 (2004)
Chen, Y., Pi, D., Wang, B.: Enhanced global flower pollination algorithm for parameter identification of chaotic and hyper-chaotic system. Nonlinear Dyn. 97(2), 1343 (2019)
Lu, Z.R., Liu, G., Liu, J., Chen, Y.M., Wang, L.: Parameter identification of nonlinear fractional-order systems by enhanced response sensitivity approach. Nonlinear Dyn. 95(2), 1495 (2019)
Narayanan, M., Narayanan, S., Padmanabhan, C.: Parametric identification of nonlinear systems using multiple trials. Nonlinear Dyn. 48(4), 341 (2007)
Marzouk, Y.M., Najm, H.N., Rahn, L.A.: Stochastic spectral methods for efficient Bayesian solution of inverse problems. J. Comput. Phys. 224(2), 560 (2007)
Marzouk, Y.M., Najm, H.N.: Dimensionality reduction and polynomial chaos acceleration of Bayesian inference in inverse problems. J. Comput. Phys. 228(6), 1862 (2009)
Brastein, O.M., Perera, D.W.U., Pfeifer, C., Skeie, N.O.: Parameter estimation for grey-box models of building thermal behaviour. Energy Build. 169, 58 (2018)
Dokos, S., Lovell, N.H.: Parameter estimation in cardiac ionic models. Prog. Biophys. Mol. Biol. 85(2–3), 407 (2004)
Kivman, G.A.: Sequential parameter estimation for stochastic systems. Nonlinear Processes Geophys. 10(3), 253 (2003)
Lorenz, E.N.: Deterministic nonperiodic flow. J. Atmos. Sci. 20(2), 130 (1963)
Springer, S., Haario, H., Shemyakin, V., Kalachev, L., Shchepakin, D.: Robust parameter estimation of chaotic systems. Inverse Probl. Imaging 13(6), 1189 (2019)
Politi, A.: Lyapunov exponent. Scholarpedia 8(3), 2722 (2013). https://doi.org/10.4249/scholarpedia.2722. Revision #137286
Govorukhin, V.: Calculation Lyapunov exponents for ode. MATLAB File Exchange (2020). Retrieved 29 June 2020
Wolf, A., Swift, J.B., Swinney, H.L., Vastano, J.A.: Determining Lyapunov exponents from a time series. Physica D 16(3), 285 (1985)
Acknowledgements
This research was primarily supported by the DARPA Physics of AI Program under the grant “Physics Inspired Learning and Learning the Order and Structure of Physics,” Agreement No. HR00111890030. It was also supported in part by the DARPA Artificial Intelligence Research Associate under the grant “Artificial Intelligence Guided Multi-scale Multi-physics Framework for Discovering Complex Emergent Materials Phenomena,” Agreement No. HR00111990028.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Pseudocode
Pseudocode
In this appendix, we provide the pseudocode for both the linear Kalman filter and nonlinear unscented Kalman filter algorithms. In the UKF algorithm, \(\alpha \) and \(\kappa \) are parameters that determine the spread of the sigma points around the mean, \(\beta \) is a parameter used for incorporating prior information on the distribution of x, and the notation \([\cdot ]_i\) denotes the i-th row of the matrix [22].
Rights and permissions
About this article
Cite this article
Galioto, N., Gorodetsky, A.A. Bayesian system ID: optimal management of parameter, model, and measurement uncertainty. Nonlinear Dyn 102, 241–267 (2020). https://doi.org/10.1007/s11071-020-05925-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11071-020-05925-8