Integrated production and maintenance planning under uncertain demand with concurrent learning of yield rate

Abstract

Strong interactions between decisions in the maintenance and production scheduling domains, and their impacts on the equipment yield rates necessitate maintenance and production decisions being optimized concurrently, with considerations of yield dependencies on the equipment conditions and production rates. This paper proposes an integrated decision-making policy for production and maintenance operations on a single machine under uncertain demand, with concurrent considerations and learning of yield dependencies on the equipment conditions and production rates. This policy is obtained through a two-stage stochastic programming model, which considers the variable demand, machine degradation, and maintenance times. This model incorporates outsourcing decisions and operational decisions regarding reworking, scraping of imperfect products to ensure the demand is adequately met. A closed-form reinforcement learning method is utilized to learn yield dependencies. Simulations confirm the necessity of yield learning and show the proposed method outperforms the traditional, fragmented approaches where the effects of production rates and machine conditions on the resulting yield rates are not considered. The two-stage stochastic setting is demonstrated by comparing with the traditional one-stage deterministic approach, where decisions are made based on the expected demand and production performance, with scrapping, reworking, and outsourcing decisions established before the demand and production performance are observed.

Appendix

In order to formulate the degradation process in the two-stage stochastic programming problem, based on the DTMC model in Sect. 2.2, constraint (4) represents a block of constraints as follows.

We use binary vectors ${{\varvec{\upxi}}}_{{{\mathbf{PRM}}}}^{{\mathbf{t}}} ,{{\varvec{\upxi}}}_{{{\mathbf{MRM}}}}^{{\mathbf{t}}} ,{{\varvec{\upxi}}}_{{{\mathbf{PM}}}}^{{\mathbf{t}}} \in {\mathbb{Z}}_{2}^{{N_{w} + N_{m} }}$ to respectively denote the uncertain perfect reactive maintenance, minimum reactive maintenance, and preventive maintenance time with elements satisfying $\sum\limits_{j = 1}^{{N_{w} }} {\xi_{*M}^{t,j} } = 0$ and $\sum\limits_{{j = N_{w} + 1}}^{{N_{w} + N_{m} }} {\xi_{*M}^{t,j} } = 1$. Element $\xi_{*M}^{t,j} = 1$ indicates that the realization of maintenance time is $(N_{m} + 1 - j)$. For example, in Eq. (14), vector ${{\varvec{\upxi}}}_{{{\mathbf{PRM}}}}^{{\mathbf{t}}}$ indicates that if perfect reactive maintenance starts to be conducted at time $t$, the realization of maintenance time is 2.

$$\begin{gathered} \,States:\,\,\begin{array}{*{20}c} {W_{1} ,W_{2} ,...,W_{{N_{w} }} }, & {M_{{N_{m} }} ...M_{2} ,M_{1} } \\ \end{array} \,\,\, \hfill \\ {{\varvec{\upxi}}}_{{{\mathbf{PRM}}}}^{{\mathbf{t}}} = \left[ {\begin{array}{*{20}c} 0 & 0 & \cdots & 0 & 0 & \cdots & 1 & 0 \\ \end{array} } \right]^{T} \hfill \\ \end{gathered}$$

(14)

We use $\Pr_{i,j}^{t}$ to denote the element of the realization of the Markovian transition matrix ${\mathbf{P}}_{{{\mathbf{trans}}}} ({\mathbf{x}},x_{PM} ,x_{RMO} )$ at time $t$. Constraint (15) defines the Failure Transition Submatrix ${\mathbf{P}}_{{\mathbf{F}}} ({\mathbf{x}},x_{RMO} )$ in the Markovian transition matrix, where $F_{qi}$ is the failure probability of the machine when it is working under production rate $q$ and degradation level $i$. This constraint aligns the vector of failure probability with the column determined by the realization of maintenance time. Other columns in the submatrix are zeros. The block of constraints (16)–(19) defines the Recovery Transition Submatrix ${\mathbf{P}}_{{\mathbf{R}}} (x_{PM} ,x_{RMO} )$. Constraint (16) defines the probability of restoring to the perfect working condition at time $t$. If we decide to perform PRM, the probability is one. In the case of MRM, if the machine is working, the probability is $\Pr_{{N_{w} + N_{m} ,1}}^{t - 1}$, if not, it equals to $s_{1}^{t}$. Constraint (17) defines the probability of restoring to other working conditions at time $t$. Constraint (18) and (19) define the initial probability of restoring to perfect and other working conditions (i.e. $t = 1$).

$$Pr_{{i,N_{w} + j}}^{t} = \left( {\sum\limits_{q = 1}^{{N_{P} }} {x_{q} F_{qi} } } \right)\left[ {\xi_{PRM}^{{t,N_{w} + j}} x_{RMO} + \xi_{MRM}^{{t,N_{w} + j}} (1 - x_{RMO} )} \right],\,\forall i \in \left[ {N_{w} } \right],\,\forall j \in \left[ {N_{m} } \right],\,\forall t \in \left[ T \right]$$

(15)

$$Pr_{{N_{w} + N_{m} ,1}}^{t} = x_{RMO} + (1 - x_{RMO} )\left[ {Pr_{{N_{w} + N_{m} ,1}}^{t - 1} \left( {1 - \sum\limits_{i^{\prime} = 1}^{{N_{w} }} {s_{i^{\prime}}^{t} } } \right) + s_{1}^{t} \sum\limits_{i^{\prime} = 1}^{{N_{w} }} {s_{i^{\prime}}^{t} } } \right],\,\forall \,t \in \left[ T \right]/\{ 1\}$$

(16)

$$Pr_{{N_{w} + N_{m} ,j}}^{t} = Pr_{{N_{w} + N_{m} ,1}}^{t - 1} \left( {1 - \sum\limits_{i^{\prime} = 1}^{{N_{w} }} {s_{i^{\prime}}^{t} } } \right) + s_{1}^{t} \sum\limits_{i^{\prime} = 1}^{{N_{w} }} {s_{i^{\prime}}^{t} } ,\,\forall j \in [N_{w} ]/\{ 1\} ,\,\forall t \in [T]/\{ 1\}$$

(17)

$$Pr_{{N_{w} + N_{m} ,1}}^{1} = s_{1}^{0} (1 - x_{PM} ) + x_{PM}$$

(18)

$$Pr_{{N_{w} + N_{m} ,j}}^{1} = s_{j}^{0} ,\,\,j \in [N_{w} ]/\{ 1\}$$

(19)

The second block of constraints (20)–(26) performs the transition of states to obtain the history of states of the machine over time $[T]$ represented by a binary matrix ${\mathbf{s}} \in {\mathbb{Z}}_{2}^{{(N_{w} + N_{m} ) \times T}}$. Constraint (20) defines the initial vector of state according to the decision for PM. Constraint (21)–(26) formulate the state transition over time $[T]$ using random variables $\xi^{t}$ as a trigger of transition at each time $t$. ${\mathbf{a}}^{{\mathbf{t}}} \in {\mathbb{Z}}_{2}^{{\left( {N_{w} + N_{m} } \right) \times \left( {N_{w} + N_{m} } \right)}}$ is a binary vector where only one element has value 1. The element $a_{i,j}^{t} = 1$ indicates that at time $t$, the machine is in state $i$ and transition trigger is larger than $\sum\limits_{j^{\prime} = 1}^{j - 1} {\Pr_{i,j^{\prime}}^{t} } ,\forall j \in [N_{w} + N_{m} ]/\{ 1\}$ or larger than 0 for $j = 1$. ${\mathbf{b}}^{{\mathbf{t}}} \in {\mathbb{Z}}_{2}^{{\left( {N_{w} + N_{m} } \right) \times \left( {N_{w} + N_{m} } \right)}}$ is a binary vector where only one element has value 1. The element $b_{i,j}^{t} = 1$ indicates that at time $t$, the machine is in state $i$ and transition trigger is smaller than or equal to $\sum\limits_{j^{\prime} = 1}^{j} {p_{i,j^{\prime}}^{t} } ,\forall j \in [N_{w} + N_{m} ]$. $M_{l\arg e}$ is a large positive number.

$${\mathbf{s}}^{{\mathbf{1}}} = {{\varvec{\upxi}}}_{{{\mathbf{PM}}}}^{{\mathbf{t}}} \,x_{PM} + {\mathbf{s}}^{{\mathbf{0}}} (1 - x_{PM} )$$

(20)

$$s_{i}^{t} \left( {\xi^{t} - \sum\limits_{j^{\prime} = 1}^{j - 1} {Pr_{i,j^{\prime}}^{t} } } \right) \le a_{i,j}^{t} M_{l\arg e} ,\,\forall i \in [N_{w} + N_{m} ],\forall j \in [N_{w} + N_{m} ]/\{ 1\} ,\,\forall t \in [T]$$

(21)

$$(a_{i,j}^{t} - 1)M_{l\arg e} \le s_{i}^{t} \left( {\xi^{t} - \sum\limits_{j^{\prime} = 1}^{j - 1} {Pr_{i,j^{\prime}}^{t} } } \right),\,\forall i \in [N_{w} + N_{m} ],\forall j \in [N_{w} + N_{m} ]/\{ 1\} ,\forall t \in [T]$$

(22)

$$a_{i,1}^{t} = s_{i}^{t} ,\forall i \in [N_{w} + N_{m} ],\forall t \in [T]$$

(23)

$$s_{i}^{t} \left( {\sum\limits_{j^{\prime} = 1}^{j} {Pr_{i,j^{\prime}}^{t} } - \xi^{t} } \right) \le b_{j}^{t} M_{l\arg e} ,\forall i,j \in [N_{w} + N_{m} ],\,\,\forall t \in [T]$$

(24)

$$(b_{j}^{t} - 1)M_{l\arg e} \le s_{i}^{t} \left( {\sum\limits_{j^{\prime} = 1}^{j} {Pr_{i,j^{\prime}}^{t} } - \xi^{t} } \right),\,\forall i,j \in [N_{w} + N_{m} ],\forall t \in [T]$$

(25)

$$s_{j}^{t} = a_{i,j}^{t - 1} \,b_{i,j}^{t - 1} ,\forall i,j \in [N_{w} + N_{m} ],\forall t \in [T]/\{ 1\}$$

(26)