An artificial neural network approach to recognise kinetic models from experimental data
Introduction
Modelling the kinetic behaviour of chemical reactions requires the construction of systems of differential and algebraic equations potentially involving a high number of state variables and kinetic parameters. The identification of kinetic models requires i) the selection of an appropriate functional form for the model equations and ii) the estimation of its kinetic parameters from experimental data (Bonvin et al., 2016). Both stages may pose significant challenges to the modeller. More specifically, there may be significant uncertainty on the relevant reactions occurring in the system and on the most appropriate functional forms for describing their dynamics. Furthermore, even if an appropriate model structure is selected, the estimation of its kinetic parameters may be impossible to perform due to identifiability problems associated to the proposed kinetic model structure (Raue et al., 2009).
A variety of tools for model validation have been proposed in the literature to leverage modelling and experimental efforts in kinetic modelling studies. Model building procedures based on data fitting start with the construction of a number of candidate model structures (Asprey and Macchietto, 2000). An identifiability analysis is then performed to evaluate if the parameters involved in the candidate models can be estimated from experimental data (Cobelli, Di Stefano, 1980, Galvanin, Ballan, Barolo, Bezzo, 2013). Models which do not pass the identifiability check are rejected at this stage. It is important to observe that even the exact model may be rejected if it does not satisfy the identifiability requirement. The kinetic parameters of the remaining models are then estimated fitting available measurements (Bard, 1974) and the fitting quality is assessed with a statistical test on the goodness-of-fit (MacKay, 1992, Silvey, 1975). Information-theoretic approaches for model selection can be employed to choose the best fitting model penalising unnecessarily complex model structures (Burnham and Anderson, 2002). Popular criteria for model selection are the Akaike information criterion (AIC) (Akaike, 1974), which lies its foundations in frequentist inference, and the Bayesian information criterion (BIC) (Schwarz, 1978). If more than one model is found adequate to represent the data, one may proceed by designing additional experiments with the aim of discriminating among the competing model structures (Buzzi-Ferraris, Forzatti, Paolo, 1990, Olofsson, Hebing, Niedenführ, Deisenroth, Misener, 2019) and then for improving parameter precision (Franceschini and Macchietto, 2008).
Whenever the selection of a physics-based kinetic model is impractical, e.g. because of an extremely high number of possible reaction pathways, one may prefer to invest modelling efforts in the identification of a data driven model. This may be any parametric model, e.g. a polynomial or a response surface (Box and Draper, 1987), which provides a convenient representation of the data (Bonvin et al., 2016). The structure of data driven models typically does not reflect the inner mechanisms of the physical system. Data driven modelling approaches represent one of the foundational paradigms in machine learning technologies (Barber, 2011, LeCun, Bengio, Hinton, 2015). Data driven models and machine learning approaches have already found successful application in many contexts in chemical engineering (Venkatasubramanian, 2019), particularly in the fields of materials design (Janet, Chan, Kulik, 2018, Hou, Dai, Wu, Chen, 1997), process operations (Lopes, Ribeiro, Reis, Silva, Portugal, Baptista, 2018, Molga, Cherbański, Szpyrkowicz, 2006, Petsagkourakis, 2020, Quadros, Reis, Baptista) and fault diagnosis (Zhao et al., 2019).
A class of machine learning models which has recently seen an increase in popularity is the Artificial Neural Network (ANN) model (Krizhevsky, Sutskever, Hinton, 2012, Russell, Norvig, 2016). The recent success of ANNs is associated primarily with 1) their flexibility in approximating any nonlinear continuous function (Hornik et al., 1989) 2) the development of efficient algorithms for ANN training (Geron, 2017, Hinton, Osindero, Teh, 2006) and 3) a steady decrease in the cost of computational power (Russell and Norvig, 2016). In chemical engineering, ANNs have been applied to address both regression and classification problems (Himmelblau, 2008, Lee, Shin, Realff, 2018). ANNs were employed for nonlinear system identification (Dua, 2011, Kramer, 1991, Petsagkourakis, 2020, Traver, Atkinson, Atkinson, 1999), model reduction (Prasad and Bequette, 2003) and process control (Bloch, Denoeux, 2003, Hussain, Kershenbaum, 2000). ANN-based classifiers have been used to support drug discovery (Wang et al., 2005), catalyst design (Goldsmith et al., 2018), reaction prediction (Coley, Jin, Rogers, Jamison, Jaakkola, Green, Barzilay, Jensen, 2019, Kayala, Baldi, 2012, Wei, Duvenaud, Aspuru-Guzik, 2016) and fault detection (Rengaswamy, Venkatasubramanian, 2000, Suewatanakal, 1993). In contrast to physics-based models, the estimation of parameters in ANNs typically requires substantial amounts of data. Furthermore, it is extremely challenging to assign physical significance to the ANNs parameters and accurate extrapolation beyond the conditions used for the identification of the ANN is generally not possible.
In this work, a novel framework for the selection of physics-based kinetic models is proposed. In the proposed framework, an ANN-based classifier is trained from in-silico experimental data with the aim of recognising the most appropriate kinetic model given the available experimental evidence. It is shown that the approach is effective for discriminating among rival model structures even when the kinetic models are not structurally identifiable. In fact, the estimation of kinetic parameters is not required in the procedure.
The present manuscript is structured as follows. A general overview on the ANN model is given in Section 2. The proposed ANN-based kinetic model recognition framework is detailed in Section 3. The framework is demonstrated on a simulated case study, which is detailed in Section 4. Results are presented and discussed in Section 5.
Section snippets
Artificial neural network classifier
Artificial Neural Networks are parametric models whose structure is loosely inspired by biological neural networks. In biological brains, a high number of interacting neural cells respond to input electrical stimuli by firing (i.e. transmitting) an electrical signal to downstream neural cells in the network. First attempts of modelling the logic behaviour of neural cells led to the development of the single layer perceptron (Rosenblatt, 1962), represented in Fig. 1a. The perceptron is a
Kinetic model recognition
We assume that a setup is available for conducting kinetic experiments on a reacting system of interest. In the setup, is an Nu × 1 array of manipulated system inputs and is an Ny × 1 array of state variables that can be sampled over time. The variable time is denoted as t. Nm potential model structures are proposed for describing the dynamic behaviour of the system:
In (4), quantities appearing with subscript l refer to the lth candidate model. More
Case study
The procedure described in Section 3 is demonstrated in this section on a simulated case study to demonstrate that ANN classifiers can successfully be employed to recognise kinetic model structures from experimental data. The performance of the proposed framework is evaluated at variable experimental conditions and different levels of measurement noise in the system.
Results
The number of nodes in the hidden layer of the ANN is optimised through grid search. The accuracy of the ANN in representing the validation set is reported in Fig. 4 as a function of the number of hidden nodes for all the cases considered in the study. In each case, the smallest number of hidden neurons associated with the highest validation accuracy is selected as the optimal number of neurons in the hidden layer.
The designed samples in case 1 are plotted in Fig. 5a. In case 1, a maximum
Final remarks
In the presented case study, the number of parameters involved in the ANN-based classifiers is substantially higher than the number of parameters involved in the candidate kinetic models. Nonetheless, it is observed that the structure of the feed-forward ANN allows for the employment of efficient training algorithms for regularised regression, e.g. the Adam, AdaGrad and RMSProp (Geron, 2017). Conversely, the estimation of parameters in kinetic models represents a case specific problem that may
Conclusion
In this work, a novel model selection framework is proposed where an Artificial Neural Network (ANN) is trained for selecting the most appropriate kinetic model given the available experimental data. The procedure starts with the formulation of a set of candidate model structures and the definition of an experimental design. The ANN is trained using experimental data generated in-silico. A dataset for ANN identification is constructed by simulating the designed experiments with all the
Symbols used
Latin symbols pre-exponential factor for the th reaction bias parameter in single layer perceptron concentration of species activation energy for the th reaction kinetic constant of the th reaction categorical variable label associated to the th element in label prediction for the th element in th element in number of available samples number of functions in the th kinetic model number of neurons in the hidden layer of the ANN number of candidate kinetic models
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
This work was supported by the Hugh Walter Stern PhD Scholarship, University College London; the Department of Chemical Engineering, University College London.
References (60)
- et al.
Statistical tools for optimal dynamic model building
Comput. Chem. Eng.
(2000) - et al.
Neural networks for process control and optimization: two industrial applications
ISA Trans.
(2003) - et al.
An improved version of a sequential design criterion for discriminating among rival multiresponse models
Chem. Eng. Sci.
(1990) - et al.
A graph-convolutional neural network model for the prediction of chemical reactivity
Chem. Sci.
(2019) A mixed-integer programming approach for optimal configuration of artificial neural networks
Chem. Eng. Res. Des.
(2010)An artificial neural network approximation based decomposition approach for parameter estimation of system of ordinary differential equations
Comput. Chem. Eng.
(2011)- et al.
Model-based design of experiments for parameter precision: state of the art
Chem. Eng. Sci.
(2008) - et al.
Multilayer feedforward networks are universal approximators
Neural Netw.
(1989) - et al.
Artificial neural network aided design of catalyst for propane ammoxidation
Appl. Catal. A
(1997) - et al.
Implementation of an inverse-model-based control strategy using neural networks on a partially simulated exothermic reactor
Chem. Eng. Res. Des.
(2000)
Machine learning: overview of the recent progresses and implications for the process systems engineering field
Comput. Chem. Eng.
Distribution models for nitrophenols in a liquid-liquid system
Chem. Eng. Sci.
Novelty detection: a review—part 1: statistical approaches
Signal Process.
Novelty detection: a review—part 2:neural network based approaches
Signal Process.
GPdoemd: a Python package for design of experiments for model discrimination
Comput. Chem. Eng.
Reinforcement learning for batch bioprocess optimization
Comput. Chem. Eng.
Nonlinear system identification and model reduction using artificial neural networks
Comput. Chem. Eng.
A fast training neural network and its updation for incipient fault detection and diagnosis
Comput. Chem. Eng.
Global-and-local-structure-based neural network for fault detection
Neural Netw.
A new look at the statistical model identification
IEEE Trans. Autom. Control
The Handbook of Brain Theory and Neural Networks
Bayesian Reasoning and Machine Learning
Nonlinear Parameter Estimation
Modeling human intuitions about liquid flow with particle-based simulation
PLoS Comput. Biol.
Linking models and experiments
Ind. Eng. Chem. Res.
Empirical Model-Building and Response Surfaces
Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach
Parameter and structural identifiability concepts and ambiguities: a critical review and analysis
Am. J. Physiol.
A general model-based design of experiments approach to achieve practical identifiability of pharmacokinetic and pharmacodynamic models
J. Pharmacokinet. Pharmacodyn.
Cited by (20)
Machine learning in chemical engineering: Hands-on activities
2024, Education for Chemical EngineersMachine learning approach for the prediction of biomass pyrolysis kinetics from preliminary analysis
2022, Journal of Environmental Chemical EngineeringCitation Excerpt :Thus, model-based predictions are generally preferred to understand and investigate the process. Among the several modeling approaches, artificial neural network (ANN) based machine learning models are getting more attention due to their better predictability of the nonlinear behavior [19,20]. They also have several other advantages of being able to model complex phenomena rapidly and easily with the versatility to perform as a black box information processor.
Autonomous chemical science and engineering enabled by self-driving laboratories
2022, Current Opinion in Chemical EngineeringCitation Excerpt :Other than individual reaction optimization and modeling, ML techniques have been utilized to perform predictions of reaction conditions for novel organic synthesis reactions, for example, by training with data from the Reaxys database [27]. The more generalized the ML predictions of reactions and process conditions are desired, the more experimental data for ML model training it requires, as the scope of possible parameters (i.e. design space of intrinsic and extrinsic parameters) becomes much larger when introducing discrete variables such as chemical species. Various data mining and in silico techniques (physics-based modeling and data augmentation [28]) can be used to supply the increased demand for data. It should be noted that in contrast to organic synthesis, the process-sensitive nature of nanomaterial syntheses limits the generalizability of the ML predictions beyond a specific class of materials.
Improving robustness of kinetic models for steam reforming based on artificial neural networks and ab initio calculations
2022, Chemical Engineering JournalOptimal Design of Experiments Based on Artificial Neural Network Classifiers for Fast Kinetic Model Recognition
2022, Computer Aided Chemical EngineeringCitation Excerpt :However, the approach was characterised by a fixed design of the experiments, which can lead to poorly informative experiments. The research here presented extends Quaglio et al. (2020) work by defining a procedure for optimal experimental design coupled to the ANN-based classifier for fast kinetic model recognition. Classical MBDoE approaches are not suitable for the direct integration of the ANN approach, because the kinetic models are not used for experimental data fitting and making predictions; therefore a new criterion for optimally designing experiments has been formulated.