Novel manifold learning based virtual sample generation for optimizing soft sensor with small data

Highlights

• A novel manifold learning based virtual sample generation method is proposed.

• Isomap is used to visualize high dimensional data for finding data sparse regions.

• Feasible virtual samples can be generated to supplement the small sample space.

• Case studies including numerical simulations and a real-world application are executed.

• The effectiveness of the proposed virtual sample generation method is conformed.

Abstract

Due to the extremely complex mechanism and strong non-linear characteristics of industrial processes, data-driven soft sensor technologies play a key role in the intelligent measurement of process industries. However, the information of the collected process data in the steady stage is quite limited and unreliable, causing the small sample problem. As a result, it becomes an intractable challenge to catch the nature of the process and build accurate soft sensor models. To solve this problem, this paper proposes a novel manifold learning based virtual sample generation method (Isomap-VSG) to generate feasible virtual samples in the information gaps for supplementing the original small sample space. To find data sparse regions reasonably, one kind of manifold learning methods called Isomap is used to visualize process data with high dimension. Then virtual samples can be generated by the interpolation method and extreme learning machine. The simulation results on a standard dataset and a real-world application demonstrate that, compared with other advanced methods, the proposed Isomap-VSG method can achieve better performance in terms of generating feasible virtual samples and improving the accuracy of soft sensor models using limited samples.

Introduction

In modern process industries, establishing accurate and reliable soft sensor models plays a crucial role in the early planning for policy makers [1], [2]. The petrochemical production process always contains a series of complex physical and chemical reactions involving multiple transfers (momentum transfer, heat transfer and mass transfer) [3], [4], [5]. Mechanism modeling requires a thorough grasp of the inherent mechanism of entire reaction processes [6]. As the process scale increases, it turns to be more difficult to establish mechanism models [7]. While, data-driven modeling approaches are primarily based on process data rather than complex mechanisms to build mathematical models [8], [9]. Data-driven modeling methods have been widely used in the establishment of soft sensor models. However, building data-driven models has specific requirements on the sample size and data characteristics. The number of valid samples is the most important factor affecting the accuracy of data-driven models [10], [11], [12]. Although advanced equipment and techniques are employed in modern process industries to collect and store large numbers of production operation data, it is still difficult to get sufficient typical data for two reasons: firstly, the scale of representative samples is limited by the steady process with small fluctuations, high cost of data collection, and low probability of abnormal events [13], [14]; secondly, it is hard to effectively extract valuable information and knowledge because the collected data may have the characteristics of non-linearity, random noise, missing values and uncertainty [15]. Due to the actual situation of insufficient sample capacity, poor representation and uneven distribution, the development and application of data-driven models is seriously constrained. These challenges can be considered as small sample problems. Small sample problems are common in many fields, such as in the field of the process industry, biomedical engineering and material science. How to establish a reliable soft-sensor model based on small data is a key issue in manufacturing management [16], [17], [18], [19]. A variety of approaches like the bat algorithm and the support vector regression (SVR) have been proposed by relevant researchers to deal with small sample problems [20], [21], [22].

Due to the limitations of imprecision, uncertainty and incompleteness of limited data, virtual sample generation (VSG) technology was proposed from the perspective of expanding sample data [23], [24]. VSG based methods can effectively expand the data quantity based on small sample data information. VSG based methods have been widely utilized to solve small sample problems and have made great progress in theoretical researches and practical applications [25], [26]. Based on the analysis of the current generation ideas, VSG based methods are usually classified into three types: sampling-based VSG, information diffusion-based VSG and feature representation-based VSG. Sampling-based VSG approaches are represented by Bootstrap and Synthetic Minority Oversampling Technique. Such methods mainly expand the sample size by resampling the small samples and understand the real distribution through the sampling distribution [27]. These methods can only expand the number of samples, but they cannot make up for the lack of small sample information. Information diffusion-based VSG methods are typified by mega trend diffusion (MTD) and tree-based trend diffusion (TTD). In [28], TTD used the information diffusion principle to derive the diffusion function for each dimension of the sample data, and then the range of the diffusion domain could be obtained. Finally, in TTD, the fuzzy theory was used to generate new samples in a certain domain range. The drawback of TTD is to calculate the expansion domain of each dimension variable separately and randomly generate virtual samples of each dimension variable without considering the correlation between variables [29]. Feature representation-based VSG methods relay on feature extraction algorithms or deep networks to find feature space suitable for processing high-dimensional samples [30], [31]. By generating virtual samples in the feature space of high-dimensional small samples, the real characteristics and real distribution can be easily learnt. Compared with sampling-based VSG and information diffusion-based VSG, feature representation-based VSG is more suitable for processing high-dimensional small data and can also generate new data to fill the information gap of small data. Thus, the work of this paper is based on the generation idea of feature representation.

Based on the generation idea of feature representation, the feature space of small samples can be learnt by feature extraction algorithms or deep networks. The latest VSG method based on deep networks is Generative Adversarial Net (GAN) in [32], [33]. GAN has been used in the field of image generation. While some problems such as mode collapse may occur in GAN. On the other hand, there is a big gap between the number of samples in the field of image processing and in the field of process industries. It is difficult to learn the parameters of deep networks from the small sample data of industrial process. The feature representation method of manifold learning called Isomap is selected to realize dimensionality reduction and the visualization of high-dimensional data, because Isomap is an effective algorithm for feature extraction. A regression model between the extracted features and the original data should be built for obtaining new samples. Extreme learning machine (ELM) as an effective non-linear mapping tool is adopted to build the regression model and the soft sensor model. To enhance the performance of soft sensor using limited data, this paper proposes a manifold learning based virtual sample generation method (Isomap-VSG). The specific steps to develop the proposed model are listed as follows: (1) Isomap is used to realize the dimensionality reduction of the original high-dimensional data. (2) Virtual samples are generated in the sparse place according to the visual structure of the data in the low-dimensional space; (3) One evaluation index for virtual samples, i.e. the range of asymmetric acceptable extension domain is calculated to select feasible virtual samples; (4) The obtained virtual samples are utilized to modify the established ELM soft sensor model. To verify the effectiveness of Isomap-VSG, case studies on a standard dataset and a real-world application are carried out. The simulation results show that the essence of the industrial process can be captured by the expanded samples and the accuracy of the soft sensor modes can be improved. Compared with SVR and several advanced VSG methods such as MTD, TTD, Bootstrap, the proposed Isomap-VSG can achieve better performance.

The specific arrangements of the remaining paper are structured as follows: Some preliminaries of ELM, manifold learning method and asymmetric acceptable domain range expansion method are described in Section 2; In Section 3, the proposed Isomap-VSG method and the specific process of soft sensor modeling are illustrated in detail; Section 4 analyzes the simulation results of a benchmark function and the Purified Terephthalic Acid production process; Section 5 draws the conclusions.