An artificial neural network approach to recognise kinetic models from experimental data

https://doi.org/10.1016/j.compchemeng.2020.106759Get rights and content

Abstract

The quantitative description of the dynamic behaviour of reacting systems requires the identification of an appropriate set of kinetic model equations. The selection of the correct model may pose substantial challenges as there may be a large number of candidate kinetic model structures. In this work, a model selection approach is presented where an Artificial Neural Network classifier is trained for recognising appropriate kinetic model structures given the available experimental evidence. The method does not require the fitting of kinetic parameters and it is well suited when there is a high number of candidate kinetic mechanisms. The approach is demonstrated on a simulated case study on the selection of a kinetic model for describing the dynamics of a three-component reacting system in a batch reactor. The sensitivity of the approach to a change in the experimental design and to a change in the system noise is assessed.

Introduction

Modelling the kinetic behaviour of chemical reactions requires the construction of systems of differential and algebraic equations potentially involving a high number of state variables and kinetic parameters. The identification of kinetic models requires i) the selection of an appropriate functional form for the model equations and ii) the estimation of its kinetic parameters from experimental data (Bonvin et al., 2016). Both stages may pose significant challenges to the modeller. More specifically, there may be significant uncertainty on the relevant reactions occurring in the system and on the most appropriate functional forms for describing their dynamics. Furthermore, even if an appropriate model structure is selected, the estimation of its kinetic parameters may be impossible to perform due to identifiability problems associated to the proposed kinetic model structure (Raue et al., 2009).

A variety of tools for model validation have been proposed in the literature to leverage modelling and experimental efforts in kinetic modelling studies. Model building procedures based on data fitting start with the construction of a number of candidate model structures (Asprey and Macchietto, 2000). An identifiability analysis is then performed to evaluate if the parameters involved in the candidate models can be estimated from experimental data (Cobelli, Di Stefano, 1980, Galvanin, Ballan, Barolo, Bezzo, 2013). Models which do not pass the identifiability check are rejected at this stage. It is important to observe that even the exact model may be rejected if it does not satisfy the identifiability requirement. The kinetic parameters of the remaining models are then estimated fitting available measurements (Bard, 1974) and the fitting quality is assessed with a statistical test on the goodness-of-fit (MacKay, 1992, Silvey, 1975). Information-theoretic approaches for model selection can be employed to choose the best fitting model penalising unnecessarily complex model structures (Burnham and Anderson, 2002). Popular criteria for model selection are the Akaike information criterion (AIC) (Akaike, 1974), which lies its foundations in frequentist inference, and the Bayesian information criterion (BIC) (Schwarz, 1978). If more than one model is found adequate to represent the data, one may proceed by designing additional experiments with the aim of discriminating among the competing model structures (Buzzi-Ferraris, Forzatti, Paolo, 1990, Olofsson, Hebing, Niedenführ, Deisenroth, Misener, 2019) and then for improving parameter precision (Franceschini and Macchietto, 2008).

Whenever the selection of a physics-based kinetic model is impractical, e.g. because of an extremely high number of possible reaction pathways, one may prefer to invest modelling efforts in the identification of a data driven model. This may be any parametric model, e.g. a polynomial or a response surface (Box and Draper, 1987), which provides a convenient representation of the data (Bonvin et al., 2016). The structure of data driven models typically does not reflect the inner mechanisms of the physical system. Data driven modelling approaches represent one of the foundational paradigms in machine learning technologies (Barber, 2011, LeCun, Bengio, Hinton, 2015). Data driven models and machine learning approaches have already found successful application in many contexts in chemical engineering (Venkatasubramanian, 2019), particularly in the fields of materials design (Janet, Chan, Kulik, 2018, Hou, Dai, Wu, Chen, 1997), process operations (Lopes, Ribeiro, Reis, Silva, Portugal, Baptista, 2018, Molga, Cherbański, Szpyrkowicz, 2006, Petsagkourakis, 2020, Quadros, Reis, Baptista) and fault diagnosis (Zhao et al., 2019).

A class of machine learning models which has recently seen an increase in popularity is the Artificial Neural Network (ANN) model (Krizhevsky, Sutskever, Hinton, 2012, Russell, Norvig, 2016). The recent success of ANNs is associated primarily with 1) their flexibility in approximating any nonlinear continuous function (Hornik et al., 1989) 2) the development of efficient algorithms for ANN training (Geron, 2017, Hinton, Osindero, Teh, 2006) and 3) a steady decrease in the cost of computational power (Russell and Norvig, 2016). In chemical engineering, ANNs have been applied to address both regression and classification problems (Himmelblau, 2008, Lee, Shin, Realff, 2018). ANNs were employed for nonlinear system identification (Dua, 2011, Kramer, 1991, Petsagkourakis, 2020, Traver, Atkinson, Atkinson, 1999), model reduction (Prasad and Bequette, 2003) and process control (Bloch, Denoeux, 2003, Hussain, Kershenbaum, 2000). ANN-based classifiers have been used to support drug discovery (Wang et al., 2005), catalyst design (Goldsmith et al., 2018), reaction prediction (Coley, Jin, Rogers, Jamison, Jaakkola, Green, Barzilay, Jensen, 2019, Kayala, Baldi, 2012, Wei, Duvenaud, Aspuru-Guzik, 2016) and fault detection (Rengaswamy, Venkatasubramanian, 2000, Suewatanakal, 1993). In contrast to physics-based models, the estimation of parameters in ANNs typically requires substantial amounts of data. Furthermore, it is extremely challenging to assign physical significance to the ANNs parameters and accurate extrapolation beyond the conditions used for the identification of the ANN is generally not possible.

In this work, a novel framework for the selection of physics-based kinetic models is proposed. In the proposed framework, an ANN-based classifier is trained from in-silico experimental data with the aim of recognising the most appropriate kinetic model given the available experimental evidence. It is shown that the approach is effective for discriminating among rival model structures even when the kinetic models are not structurally identifiable. In fact, the estimation of kinetic parameters is not required in the procedure.

The present manuscript is structured as follows. A general overview on the ANN model is given in Section 2. The proposed ANN-based kinetic model recognition framework is detailed in Section 3. The framework is demonstrated on a simulated case study, which is detailed in Section 4. Results are presented and discussed in Section 5.

Section snippets

Artificial neural network classifier

Artificial Neural Networks are parametric models whose structure is loosely inspired by biological neural networks. In biological brains, a high number of interacting neural cells respond to input electrical stimuli by firing (i.e. transmitting) an electrical signal to downstream neural cells in the network. First attempts of modelling the logic behaviour of neural cells led to the development of the single layer perceptron (Rosenblatt, 1962), represented in Fig. 1a. The perceptron is a

Kinetic model recognition

We assume that a setup is available for conducting kinetic experiments on a reacting system of interest. In the setup, u is an Nu × 1 array of manipulated system inputs and y is an Ny × 1 array of state variables that can be sampled over time. The variable time is denoted as t. Nm potential model structures are proposed for describing the dynamic behaviour of the system:fl(x˙l,xl,u,t,θl)=0y^l=hl(xl)l=1,,Nm

In (4), quantities appearing with subscript l refer to the lth candidate model. More

Case study

The procedure described in Section 3 is demonstrated in this section on a simulated case study to demonstrate that ANN classifiers can successfully be employed to recognise kinetic model structures from experimental data. The performance of the proposed framework is evaluated at variable experimental conditions and different levels of measurement noise in the system.

Results

The number of nodes in the hidden layer of the ANN is optimised through grid search. The accuracy of the ANN in representing the validation set is reported in Fig. 4 as a function of the number of hidden nodes for all the cases considered in the study. In each case, the smallest number of hidden neurons associated with the highest validation accuracy is selected as the optimal number of neurons in the hidden layer.

The designed samples in case 1 are plotted in Fig. 5a. In case 1, a maximum

Final remarks

In the presented case study, the number of parameters involved in the ANN-based classifiers is substantially higher than the number of parameters involved in the candidate kinetic models. Nonetheless, it is observed that the structure of the feed-forward ANN allows for the employment of efficient training algorithms for regularised regression, e.g. the Adam, AdaGrad and RMSProp (Geron, 2017). Conversely, the estimation of parameters in kinetic models represents a case specific problem that may

Conclusion

In this work, a novel model selection framework is proposed where an Artificial Neural Network (ANN) is trained for selecting the most appropriate kinetic model given the available experimental data. The procedure starts with the formulation of a set of candidate model structures and the definition of an experimental design. The ANN is trained using experimental data generated in-silico. A dataset for ANN identification is constructed by simulating the designed experiments with all the

Symbols used

Latin symbols
Ajpre-exponential factor for the jth reaction
bbias parameter in single layer perceptron
Ciconcentration of species i
Ea,jactivation energy for the jth reaction
kjkinetic constant of the jth reaction
lcategorical variable
lilabel associated to the ith element in Ψ
l^ilabel prediction for the ith element in Ψ
n^i,jjth element in n^i
Nnumber of available samples
Nf,lnumber of functions in the lth kinetic model
Nhnumber of neurons in the hidden layer of the ANN
Nmnumber of candidate kinetic models

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This work was supported by the Hugh Walter Stern PhD Scholarship, University College London; the Department of Chemical Engineering, University College London.

References (60)

  • J.H. Lee et al.

    Machine learning: overview of the recent progresses and implications for the process systems engineering field

    Comput. Chem. Eng.

    (2018)
  • A.L.C.V. Lopes et al.

    Distribution models for nitrophenols in a liquid-liquid system

    Chem. Eng. Sci.

    (2018)
  • M. Markou et al.

    Novelty detection: a review—part 1: statistical approaches

    Signal Process.

    (2003)
  • M. Markou et al.

    Novelty detection: a review—part 2:neural network based approaches

    Signal Process.

    (2003)
  • S. Olofsson et al.

    GPdoemd: a Python package for design of experiments for model discrimination

    Comput. Chem. Eng.

    (2019)
  • PetsagkourakisP. et al.

    Reinforcement learning for batch bioprocess optimization

    Comput. Chem. Eng.

    (2020)
  • V. Prasad et al.

    Nonlinear system identification and model reduction using artificial neural networks

    Comput. Chem. Eng.

    (2003)
  • R. Rengaswamy et al.

    A fast training neural network and its updation for incipient fault detection and diagnosis

    Comput. Chem. Eng.

    (2000)
  • H. Zhao et al.

    Global-and-local-structure-based neural network for fault detection

    Neural Netw.

    (2019)
  • H. Akaike

    A new look at the statistical model identification

    IEEE Trans. Autom. Control

    (1974)
  • M.A. Arbib

    The Handbook of Brain Theory and Neural Networks

    (2003)
  • D. Barber

    Bayesian Reasoning and Machine Learning

    (2011)
  • Y. Bard

    Nonlinear Parameter Estimation

    (1974)
  • BatesC.J. et al.

    Modeling human intuitions about liquid flow with particle-based simulation

    PLoS Comput. Biol.

    (2019)
  • D. Bonvin et al.

    Linking models and experiments

    Ind. Eng. Chem. Res.

    (2016)
  • G.E.P. Box et al.

    Empirical Model-Building and Response Surfaces

    (1987)
  • K.P. Burnham et al.

    Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach

    (2002)
  • Chollet, F., et al., 2015. Keras....
  • C. Cobelli et al.

    Parameter and structural identifiability concepts and ambiguities: a critical review and analysis

    Am. J. Physiol.

    (1980)
  • F. Galvanin et al.

    A general model-based design of experiments approach to achieve practical identifiability of pharmacokinetic and pharmacodynamic models

    J. Pharmacokinet. Pharmacodyn.

    (2013)
  • Cited by (20)

    • Machine learning approach for the prediction of biomass pyrolysis kinetics from preliminary analysis

      2022, Journal of Environmental Chemical Engineering
      Citation Excerpt :

      Thus, model-based predictions are generally preferred to understand and investigate the process. Among the several modeling approaches, artificial neural network (ANN) based machine learning models are getting more attention due to their better predictability of the nonlinear behavior [19,20]. They also have several other advantages of being able to model complex phenomena rapidly and easily with the versatility to perform as a black box information processor.

    • Autonomous chemical science and engineering enabled by self-driving laboratories

      2022, Current Opinion in Chemical Engineering
      Citation Excerpt :

      Other than individual reaction optimization and modeling, ML techniques have been utilized to perform predictions of reaction conditions for novel organic synthesis reactions, for example, by training with data from the Reaxys database [27]. The more generalized the ML predictions of reactions and process conditions are desired, the more experimental data for ML model training it requires, as the scope of possible parameters (i.e. design space of intrinsic and extrinsic parameters) becomes much larger when introducing discrete variables such as chemical species. Various data mining and in silico techniques (physics-based modeling and data augmentation [28]) can be used to supply the increased demand for data. It should be noted that in contrast to organic synthesis, the process-sensitive nature of nanomaterial syntheses limits the generalizability of the ML predictions beyond a specific class of materials.

    • Optimal Design of Experiments Based on Artificial Neural Network Classifiers for Fast Kinetic Model Recognition

      2022, Computer Aided Chemical Engineering
      Citation Excerpt :

      However, the approach was characterised by a fixed design of the experiments, which can lead to poorly informative experiments. The research here presented extends Quaglio et al. (2020) work by defining a procedure for optimal experimental design coupled to the ANN-based classifier for fast kinetic model recognition. Classical MBDoE approaches are not suitable for the direct integration of the ANN approach, because the kinetic models are not used for experimental data fitting and making predictions; therefore a new criterion for optimally designing experiments has been formulated.

    View all citing articles on Scopus
    View full text