Risk mitigation in model-based experiment design: A continuous-effort approach to optimal campaigns

https://doi.org/10.1016/j.compchemeng.2022.107680Get rights and content

Highlights

  • Novel bi-objective formulation to mitigate non-informative experiments in model-based experimental design under uncertainty.

  • Conditional-value-at-risk is considered alongside the average information in a bi-objective optimization problem.

  • Discretization of experimental set combined with continuous-effort approximation yield convex optimization regardless of model structure.

  • Characterization of a Pareto-efficient frontier from which experimenters can choose depending on their attitude to risk.

  • Industrially relevant experiment design problems proven computationally tractable with the proposed methodology.

Abstract

A key challenge in maximizing the effectiveness of model-based design of experiments for calibrating nonlinear process models is the inaccurate prediction of information that is afforded by each new experiment. We present a novel methodology to exploit prior probability distributions of model parameter estimates in a bi-objective optimization formulation, where a conditional-value-at-risk criterion is considered alongside an average information criterion. We implement a tractable numerical approach that discretizes the experimental design space and leverages the concept of continuous-effort experimental designs in a convex optimization formulation. We demonstrate effectiveness and tractability through three case studies, including the design of dynamic experiments. In one case, the Pareto frontier comprises experimental campaigns that significantly increase the information content in the worst-case scenarios. In another case, the same campaign is proven to be optimal irrespective of the risk attitude. An open-source implementation of the methodology is made available in the Python software Pydex.

Introduction

Optimal experimental design (OED) is a powerful paradigm for computing maximally informative experimental campaigns. The seminal example is its application in the response surface methodology (RSM) first described in Box and Wilson (1951). It led to the discovery of a set of standard designs that are optimal for calibrating RSM models. These are the celebrated factorial (experimental campaign) designs (Box and Wilson, 1951) and their derivatives, namely fractional factorial (Box, Hunter, 1961) , Box-Behnken (Box, Behnken, 1960), Box-Wilson’s central composite (Box, Wilson, 1951), and Plackett-Burman (Plackett, Burman, 1946). RSM models have furthermore been extended to dynamic systems and experimentation (Georgakis, 2013, Wang, Georgakis, 2017). This popularity hinges on the fact that experimental designs for linear models, like RSMs, are independent of the unknown model parameters making them inherently resilient to parametric uncertainty and thus reproducible. But there are also some limitations of RSM. For large and complex systems, even low-order polynomial approximants can require an impractical number of experiments to develop.

An alternate philosophy is the model-based framework whereby the focus is on developing mechanistic mathematical models, which have superior extrapolative capabilities. To emphasize this distinction, the term model-based design of experiments (MBDoE) has been coined in the Process Systems Engineering (PSE) community; see for instance, the seminal survey article by Franceschini and Macchietto (2008).

One added complexity in MBDoE is the nonlinear nature of mechanistic models. Unlike linear models, there does not exist a set of standard designs for nonlinear models because the optimal designs are now dependent on the values specified for the model parameters. In other words, MBDoE provides tailored solutions to a given mechanistic model and a given set of parameter values. This necessitates an experimenter to (re-)compute the optimal design each time the model structure or its parameter values are updated.

The basic technique to overcome non-linearity is local design, whereby the experimenter selects a set of nominal model parameter values. Experiments are then designed for the linearized version of the model, constructed around these nominal parameter values. A drawback is that a local design may lead to uninformative or even infeasible experiments when inaccurate nominal parameter values are used. Addressing the former issue is the main focus of this article.

An improvement to local designs is the sequential experimentation strategy, whereby (local) experiment design and parameter estimation are conducted iteratively, until a desired model parameter precision is obtained (Franceschini and Macchietto, 2008). Frequent parameter updates are made in hopes that the parameter values progressively get more accurate, thereby increasing the overall information content. The success of the sequential strategy has been showcased in many applications; see Stigter et al. (2006), Galvanin et al. (2009), and Cruz Bournazou et al. (2017), to name a few. Note that there is flexibility in how frequently the model parameters are updated, that is, how many experimental runs are conducted before a parameter estimation is repeated. In Galvanin et al. (2009), parameter updates are done as soon as new measurements are acquired. As a result, experiments for dynamic systems can be re-designed to the most updated optimal trajectory on the fly. There is furthermore no restriction on the number of experiments that are executed in parallel. In Cruz Bournazou et al. (2017), the redesign principle of Galvanin et al. (2009) is demonstrated in a parallel experimental setup, where multiple experiments are run and re-designed simultaneously.

A second class of MBDoE methods modify the experimental information criterion to make it more resilient to uncertain model parameters. These include robust MBDoE methods (Asprey et al., 2000, Bazil, Buzzard, Rundell, 2012). Although they are rarely used together (due to computational complexity), robust criteria could be combined with the sequential/redesign strategy. Two popular robust criteria are the expected value approach and the worst-case approach (Asprey et al., 2000, Asprey, Macchietto, 2002). The former is also referred to as the mean value approach (Pázman and Pronzato, 2007) or the pseudo-Bayesian approach (Ryan et al., 2015). It requires a prior probability distribution on the unknown model parameter values to be specified, e.g., obtained as the result of a Bayesian parameter estimation exercise. The worst-case approach is also known as the maximin formulation and seeks to maximize the information content in the worst case of scenarios (Körkel, Kostina, Bock, Schlöder, 2004, Pronzato, Walter, 1988, Telen et al., 2012). It requires a bounded set of parameter values to be specified e.g., obtained as confidence bounds from a frequentist parameter estimation or as the outcome of a set-membership parameter estimation (Gottu Mukkula, Paulen, 2017, Jaulin, Walter, 1993, Pankajakshan, Quaglio, Galvanin, 2018, Paulen, Gomoescu, Chachuat, 2020, Perić et al., 2018). An alternative robust approach which uses Sobol indices (Sobol, 2001) in place of the local sensitivities has also been proposed in Rodriguez-Fernandez et al. (2006). But follow-up investigations revealed that such a one-to-one exchange of local sensitivities for Sobol indices could be problematic in practice (Chu, Hahn, 2013, Schenkendorf, Xie, Rehbein, Scholl, Krewer, 2018, Abt, Barz, Cruz-Bournazou, Herwig, Kroll, Mller, Prtner, Schenkendorf, 2018).

Pázman and Pronzato (2007) highlighted the challenges that may arise with the average and maximin criteria. A key challenge with the average criterion is that it can be overly optimistic (risk-inclined) by providing a highly informative design for some parameter scenarios, but one that is also unacceptably uninformative for a significant percentage of parameter scenarios. The maximin approach does not share the overly optimistic issue, but faces other challenges. Maximin designs are quite sensitive to the choice of the bounded set given because the worst parameter scenario typically happens on the boundary of this set. Additionally, they are more prone to facing issues with singular information matrices because the worst-case scenario is by definition low in information. Lastly, one may regard the maximin approach as being overly pessimistic (risk-averse), potentially sacrificing relatively large losses in average information content for a relatively inconsequential hedge against the worst-case scenario.

In response to this, Pázman and Pronzato (2007) introduced the quantile criteria, which seek to maximize the worst-case information after excluding all those designs whose information content is below a given percentile. These criteria are akin to a value-at-risk in portfolio optimization and have the main drawback of not being concave in general. By contrast, the conditional-value-at-risk (CVaR) criterion (Valenzuela et al., 2015) maximizes the expected information content for all parameter scenarios with information content below a given percentile, which is a concave measure of risk and amenable to a linear formulation in optimization problems (Rockafellar and Uryasev, 2000). Properties of the CVaR criterion have been further investigated in Sternmllerová (2018), including its relationship to the average and maximin approaches.

In this article, we propose a novel approach to using the CVaR criterion as an additional objective alongside the average criterion. A key benefit of this bi-objective framework is the ability to describe the trade-off between risk-inclined and risk-averse attitudes as a set of efficient campaigns. In this manner, the experimenter is informed of any campaigns which might significantly reduce the risk of low information content when things are not as expected, at the cost of a relatively small decrease in average experimental information.

The rest of the paper begins with a formal problem statement of the bi-objective experimental design problem. A description and discussion of the novel methodology and numerical solution approach follows. This methodology is applied to case studies of increasing complexity to demonstrate its effectiveness and tractability, before concluding the paper.

Section snippets

Problem statement and methodology

Consider a multi-input multi-response model of a process given byy=f(x,θ)+ϵ, where xXRnx is the vector of experimental controls, yYRny the vector of responses, θΘRnθ the vector of model parameters, f:Rnx×RnθRny the process model, and ϵRny the vector of measurement errors. For simplicity the errors ϵ are assumed to follow a distribution with zero mean E[ϵ]=0 and given covariance matrix Var[ϵ]=Σy, although this is not a limitation of methodology.

The model is to be calibrated with a set of

Computational framework

Solving the bi-objective experimental design problem (13) can be challenging, even for simple process models. The proposed computational framework builds on the continuous-effort design (CED) method (Fedorov, Leonov, 2014, Kusumo et al., 2021, Vanaret, Seufert, Schwientek, Karpov, Ryzhakov, Oseledets, Asprion, Bortz, 2021) in combination with the ϵ-constraint method (Chankong, Haimes, 1983, Miettinen, 2012). The prior distribution of the uncertain model parameters is furthermore discretized by

Case studies

We start with showcasing the method on a simple algebraic model. The simplicity of this example is instrumental to illustrate some of the key features of the bi-objective designs. In particular, we put some emphasis on understanding what makes an optimistic design different to a risk-averse design for the model.

We then move on to a small-size dynamic experimental design problem involving a fed-batch reactor. The focus is on highlighting the suitability of the method to dynamic experimental

Conclusions

Many challenges faced in computing optimal experimental designs for nonlinear process models are rooted in the inconvenient fact that they are dependent on the unknown model parameter values. When unsuitable initial guesses are used, computed optimal experiments may turn out to be uninformative or infeasible.

We highlighted the importance and some developments in the literature on the issue of sub-optimality, focusing on methods that modify the information criteria to a more appropriate one. We

CRediT authorship contribution statement

Kennedy Putra Kusumo: Conceptualization, Methodology, Software, Formal analysis, Visualization, Writing – original draft. Kamal Kuriyan: Conceptualization, Methodology, Software, Writing – review & editing. Shankarraman Vaidyaraman: Conceptualization, Writing – review & editing. Salvador García-Muñoz: Conceptualization, Writing – review & editing. Nilay Shah: Conceptualization, Writing – review & editing, Supervision, Funding acquisition. Benoît Chachuat: Conceptualization, Methodology, Writing

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This work is co-funded by Eli Lilly & Company through the Pharmaceutical Systems Engineering Lab (PharmaSEL) and by the Engineering and Physical Sciences Research Council (EPSRC) as part of its Prosperity Partnership Programme under grant EP/T518207/1. We thank the three anonymous reviewers whose meticulous comments and suggestions helped improve this work.

References (50)

  • F. Pukelsheim et al.

    Efficient rounding of approximate designs

    Biometrika

    (1992)
  • K. Sternmllerová

    Optimum Design in Nonlinear Models

    (2018)
  • S. Vajda et al.

    Qualitative and quantitative identifiability analysis of nonlinear chemical kinetic models

    Chem. Eng. Commun.

    (1989)
  • C. Vanaret et al.

    Two-phase approaches to optimal model-based design of experiments: how many experiments and which ones?

    Comput. Chem. Eng.

    (2021)
  • Z. Wang et al.

    New dynamic response surface methodology for modeling nonlinear processes over semi-infinite time horizons

    Ind. Eng. Chem. Res.

    (2017)
  • A. Agrawal et al.

    A rewriting system for convex optimization problems

    J. Control Decis.

    (2018)
  • E.D. Andersen et al.

    The MOSEK interior point optimizer for linear programming: an implementation of the homogeneous algorithm

    High Performance Optimization

    (2000)
  • J.A.E. Andersson et al.

    CasADi—A software framework for nonlinear optimization and optimal control

    Math. Program. Comput.

    (2019)
  • P. Artzner et al.

    Coherent measures of risk

    Math. Finance

    (1999)
  • S.P. Asprey et al.

    Designing robust optimal dynamic experiments

    Journal of Process Control

    (2002)
  • J.N. Bazil et al.

    A global parallel model based design of experiments method to minimize model output uncertainty

    Bull. Math. Biol.

    (2012)
  • G.E. Box et al.

    Some new three level designs for the study of quantitative variables

    Technometrics

    (1960)
  • G.E. Box et al.

    The 2kp fractional factorial designs

    Technometrics

    (1961)
  • G.E.P. Box et al.

    On the experimental attainment of optimum conditions

    J. R. Stat. Soc.

    (1951)
  • V. Chankong et al.

    Optimization-based methods for multiobjective decision-making-an overview

    Large Scale Syst. Inf. Decis. Technol.

    (1983)
  • Cited by (0)

    View full text