Automated search of process control limits for fault detection in time series data

Highlights

• Summary of existing time series classification methods and their practical feasibility for productive deployment.

• Presentation of the algorithm including a brief introduction into the theoretical fundamentals of the underlying statistical techniques.

• Experimental evaluation and validation of the algorithm on public datasets and real-world manufacturing data.

• Publication of the source code to facilitate adoption by practitioners and researchers.

Abstract

Manually defined control limits remain a common strategy for quality control in manufacturing due to their ease of deployment on the shop floor compared to more advanced data analysis approaches. Despite their continued importance, there is no systematic method of defining these control limits. However, sub-optimal control limits can lead to undetected faults or cause unnecessary interruption to production. This manuscript presents an algorithm that systematizes this manual process into an efficient search task. We conceptualized the search task as a sequence of sub-problems that are based on the conventional steps taken by process experts when defining control limits. This algorithm can be integrated into an expert tool for shop floor personnel to automate the definition of control limits in annotated time series data. We demonstrate the efficacy of the control limits found by our algorithm by comparing them to those manually defined by process experts in real-world process data from the automotive industry. Furthermore, we show that our algorithm generalizes to traditional time series classification problems and achieves state-of-the-art performance on selected benchmark datasets. Our work is the first effort in automating the otherwise manual definition of control limits for fault detection.

Introduction

Control limits are a common process monitoring approach in the manufacturing domain to preemptively detect quality faults. Specifically, control limits raise a warning if the process deviates from a predefined normal state. Often, statistical control limits are used as part of a broader methodology known as statistical process control (SPC) [1], [2]. Statistical control limits are derived from the variation of the process and used for basic quality control. A process is considered under control if it lies within these control limits and exhibits minimal variation over time. Instead, values laying outside statistical control limits indicate the presence of an uncontrolled influence on the process. Aside from statistical process control limits, process experts may manually define additional control limits (i.e., empirical control limits) to detect the occurrence of a specific fault. This is especially relevant for the monitoring of time series process data, as it is difficult to generalize statistical process control to time series data  [3], [4], [5], [6]. A real-world example is depicted in Fig. 1, where empirical control limits are used to detect a verified manufacturing fault in an otherwise under-control tightening process.

Effectively, process experts use empirical control limits to perform a one-vs-all classification of time series data. This is not ideal, as empirical control limits merely detect if a process behaves differently to a defined (normal) state A. Instead, the objective should be to be warned if a process behaves similar to a particular abnormal state B. There is a wide range of classification algorithms to classify time series directly, ranging from statistical modeling to machine learning and deep learning [7]. Researchers have empirically demonstrated the applicability of these methods for numerous real-world applications [8]. Nevertheless, empirical control limits remain widespread in practice for fault detection [9]. This is largely motivated by their ease of implementation and deployment. Training and validating a classification model is often an iterative process in practice, that requires both time and expertise on the part of the model developer. In contrast, empirical control limits are easily implemented in existing process monitoring and control systems through a familiar human–machine interface. Thus, they can be deployed as an immediate countermeasure in response to a quality issue while remedial action is underway. Furthermore, the direct deployment of a model to production machines is not always possible due to security concerns and hardware constraints [10], [11]. A real-time capable model-based warning system often requires additional infrastructure. In contrast, the infringement of empirical control limits raises a real-time warning without requiring any additional investment.

Empirical control limits continue to be highly relevant as a standard tool for fault detection. They enable a fast and effective response from shop floor personnel. The high practical relevance would suggest that established tools exist to automatically find empirical control limits for a given defect. At the very least, one would expect the manual process of searching for these control limits to follow a systematic methodology. However, there are no empirical studies or systematic research in this area. This is remarkable, considering the drawbacks of sub-optimal empirical control limits. Conservative control limits can cause several processes to be incorrectly classified as faulty (i.e., false positives). Such misclassification results in unnecessary rework or scrap and could severely degrade machine productivity. On the other hand, lax control limits can fail to detect several faults (i.e., false negatives).

Setting empirical control limits is a delicate balancing act. Process experts often rely on feedback from the shop floor to adjust the control limits if either the false positive or false negative rate is unacceptably high. Also, they may conduct experiments to replicate the underlying fault and collect additional data to set more effective control limits. An algorithm that uses all available data to find empirical control limits can substantially reduce the associated effort, shorten the feedback cycle or even avoid it all together.

The key contribution of the work presented in this manuscript is the automation of the manual process of defining control limits in time series data. To the best of our knowledge, there is no dedicated literature concerned with automating this task. The algorithm is comprised of a sequence of steps that are loosely inspired by the steps performed by process experts when manually defining such control limits. The manual process is broken down into sub-problems. The algorithm uses statistical techniques to solve these sub-problems individually — the final control limits are an aggregate result of these solutions. This top-down approach results in an efficient search algorithm that can be easily automated. The user can adjust the minimum precision to return control limits that are more suitable for a particular fault detection strategy. The algorithm is validated using real-world process data from the automotive industry as well as selected benchmark datasets. The former shows its applicability for the intended task i.e. fault detection in the manufacturing domain while the latter serves to illustrate its generalizability. For the real-world data, the control limits found by the algorithm show slightly better performance than those manually defined by process experts. For the benchmark data, the algorithm matches state-of-the-art performance.

To summarize, this paper makes the following contributions to the current state-of-the-art:

• Analysis and systematization of the steps performed by process experts when manually defining control limits in practice.

• Presentation a novel algorithm that automates the search for control limits for fault detection in time series data.

• Validation of the algorithm using quality faults in real-world process data from the manufacturing domain. We show that control limits found by the algorithm are at least as effective as those manually defined by process experts.

• We demonstrate the scalability of the algorithm for traditional time series classification by testing it on selected benchmark datasets.

The rest of the paper is structured as follows: in Section 2, we introduce the concept of control limits for fault detection and provide an overview of applicable approaches. We present the theoretical background for the definition of control limits in Section 3. In Section 4, we describe the implementation of the proposed algorithm. In Section 5, we contrast its performance with manually defined control limits for real-world data. Additionally, we evaluate its performance for traditional time-series classification. We discuss the applicability and limitations of the algorithm in Sections 6 Discussion, 7 Conclusion.