1.

Introduction

Inspection of the solder joints on a printed circuit board (PCB) is challenging because such defects cannot be observed directly due to the solder joints being sandwiched between the PCB and an integrated circuit (IC) chip. To solve this problem, automated x-ray inspection, which can perform nondestructive inspection, is generally employed.^1,2 In our method, we employed an automated x-ray inspection that collects sliced images of the solder joints by x-ray computed tomography (CT) scans on the x-ray inspection machine and detects defects in the solder joints.

In recent years, automatic visual inspection systems using machine learning, especially deep learning, have been studied as a method of classifying normal and anomalous samples. This is motivated by the fact that inspection by human experts is problematic, with fatigue possibly causing the expert to miss anomalous samples. One of the most popular anomaly detection methods using machine learning is a one-class support vector machine (OCSVM).³ This method requires handcrafted features extracted by human experts in advance. Then, the extracted features are input to the trained OCSVM, and inputs are classified by the output of OCSVM. In this case, OCSVM is trained with only normal samples, but it has the disadvantage of the feature needing to be designed by human experts in advance and requiring redesign of the feature extraction method when the product specification is changed. When deep learning methods are used, because product images are directly inputted to neural networks, extracting features by human experts is not required. Therefore, even if the product specification is changed, only network retraining is required; thus the operating cost can be greatly reduced. In general, one of the anomaly detection methods using deep learning is to classify normal and anomalous samples using a binary classifier.^4,5 However, in anomaly detection for industrial products, it is difficult to guarantee enough anomalous product samples for training the classifier because defects rarely occur on the production line. Therefore, anomaly detection is generally performed using only normal data.^3,6 However, because a small number of anomalous samples is sometimes available for the learning phase, improvement of performance can be expected by adding anomalous samples to the training dataset. In this method, normal samples as well as a small number of anomalous samples were used for learning. In particular, our method extracts features following the standard Gaussian distribution by an adversarial autoencoder (AAE)⁷ from such imbalanced samples. Furthermore, anomaly scores are calculated from the features by Hotelling’s T-squared method⁸ and each solder joint is classified by an anomaly score threshold. In this experiment, we show that our method is superior to the method using handcrafted features and OCSVM on the imbalanced samples. Our contribution is a method that detects defects from a large number of normal samples and a small number of anomalous samples during the quality inspection of industrial products.

2.

Related Work

Recently, the x-ray CT method has been mainly used to detect anomalies in PCB solder joints because they cannot be observed directly. The x-rays pass through the PCB because it consists of materials with low atomic weight, but solder joints are imaged because they have high atomic weight.⁹ For example, the solder ball portion of the solder joints is represented as voxel data to obtain the condition of the solder joints using two-dimensional x-ray CT images taken from multiple directions.¹⁰ The voxel data are input to a three-dimensional convolutional neural network and classified by the output of the network. However, in typical anomaly detection tasks, a neural network classifier has the problem of requiring both normal and anomalous samples for the training stage, and their prediction performance is unstable for unknown anomalous samples not seen in training samples. Therefore, training methods that can produce satisfactory classification results when only normal samples or a small number of anomalous samples are used are needed.

A previously developed anomaly detection method uses an OCSVM in the latent space of extracted features. However, this has some disadvantages. The feature extraction method must be designed beforehand, and the features are changed by every target. To solve this problem, an autoencoder,¹¹ which is a model of a neural network, extracts the features in the latent space from the input samples automatically. In anomaly detection methods using an autoencoder, methods based on reconstruction error⁶ and the normal condition model in the latent space¹² are used. The proposed method belongs to the latter approach. In the former method, the networks are usually trained with only normal samples. As a consequence, the networks can reconstruct normal samples with small reconstruction errors; however, anomalous samples cannot be reconstructed, and the reconstruction errors become large. Hence, the samples are classified by a threshold for reconstruction errors. In the latter method, a normal model is defined in the latent space, and the likelihood of an input sample being in this space is calculated to classify it. In Ref. 12, test samples are classified by a threshold not only for the reconstruction error but also for the likelihood for a Gaussian distribution of the features extracted by the AAE. Compared with our method, it is different in terms of thresholding on the reconstruction error and likelihood, rather than on anomaly scores calculated by Hotelling’s T-squared method.

3.

Proposed Method

3.1.

X-Ray Computed Tomography

Because the solder joints sandwiched between the PCB and the IC chip cannot be inspected directly, we obtain sliced images of the solder joints with x-ray CT. When the IC chip and the PCB are joined, many solder joints are formed. Our approach is to detect each solder joint and cut out these places in advance to capture sliced images of each solder joint. The number of sliced images $\lambda$ is taken from each solder joint, and we define these sliced images as the sample for one solder joint. Hence, only anomalous solder joints can be treated as anomalous samples in each PCB where anomalous solder joints exist. An overview of the method for capturing sliced images of a solder joint is shown in Fig. 1. We took eight sliced images from one solder joint. Each is assigned a layer number corresponding to its image layer.

Fig. 1

Overview of obtaining sliced images with x-ray CT. Each image is given a layer number.

Examples of the captured sliced images of normal samples are shown in Fig. 2(a), and Fig. 2(b) shows anomalous samples.

Fig. 2

Examples of eight sliced images for normal and anomalous solder joints. Black dots in some normal sliced images are not anomalous factors. Anomalies are indicated by solder joints that are thinner than normal.

3.2.

Hotelling’s T-Squared Method

The number of anomalous samples is much smaller than the number of normal samples; thus, the normal model is defined from only normal samples or a small number of anomalous samples. If it is assumed that the normal model generated from the dataset $Z=({\mathbf{z}}_{\mathbf{1}},{\mathbf{z}}_{\mathbf{2}},\cdots ,{\mathbf{z}}_n)$ and each $\mathbf{z}=(z_1,z_2,\cdots ,z_d)\in {\mathbb{R}}^d$ is represented by the parameter $\mathbf{\theta }$, the negative log-likelihood probability $a({\mathbf{z}}^{\prime })$ of unknown sample ${\mathbf{z}}^{\prime }$ is defined as an anomaly score in the following equation:

Eq. (1)

$$a({\mathbf{z}}^{\prime })=-\log q({\mathbf{z}}^{\prime }|\mathbf{\theta }).$$

In the normal model $q(\mathbf{z}|\mathbf{\theta })$, the probability density of the normal samples is high and that of the anomalous samples is low. Therefore, the anomaly scores of the former are low and those of the latter are high, and it is possible to classify normal and anomalous samples by a threshold on the anomaly score. Hotelling’s T-squared distribution is an anomaly detection method that can be applied to a dataset following a Gaussian distribution. Here, $a({\mathbf{z}}^{\prime })$ of ${\mathbf{z}}^{\prime }$ is calculated as Eq. (2) using the two parameters of the Gaussian distribution, latent vector $\mathbf{\mu }$ and variance-covariance matrix $\mathbf{\Sigma }$:

Eq. (2)

$$\begin{gathered} a({\mathbf{z}}^{\prime })=-\log \mathcal{N}({\mathbf{z}}^{\prime }|\mathit{\mu },\mathbf{\Sigma }) \\ =-\log \frac{1}{\sqrt{{(2\pi )}^d|\mathbf{\Sigma }|}}\exp (-\frac{1}{2}{({\mathbf{z}}^{\prime }-\mathit{\mu })}^{⊺}{\mathbf{\Sigma }}^{-1}({\mathbf{z}}^{\prime }-\mathit{\mu })) \\ \propto {({\mathbf{z}}^{\prime }-\mathit{\mu })}^{⊺}{\mathbf{\Sigma }}^{-1}({\mathbf{z}}^{\prime }-\mathit{\mu }). \end{gathered}$$

The last term of Eq. (2) is equal to the Mahalanobis distance. Moreover, if $\mathit{\mu }=\mathbf{0}$ and $\mathbf{\Sigma }=\mathbf{I}$, the dataset follows the standard Gaussian distribution, and $a({\mathbf{z}}^{\prime })$ is calculated by the following equation:

Eq. (3)

$$\begin{gathered} a({\mathbf{z}}^{\prime })=-\log \mathcal{N}(\mathbf{z}|\mathbf{0},\mathbf{I}) \\ =-\log \frac{1}{\sqrt{{(2\pi )}^d}}\exp (-\frac{1}{2}{\mathbf{z}}^{\prime ⊺}{\mathbf{z}}^{\prime }) \\ \propto {\mathbf{z}}^{\prime ⊺}{\mathbf{z}}^{\prime }. \end{gathered}$$

The last term of Eq. (3) is equal to the Euclidean distance. In Hotelling’s T-squared method, $a({\mathbf{z}}^{\prime })$ follows the chi-square distribution with the degree of freedom of $d$ and the scale factor of 1. The chi-square distribution with $d=16$ is shown in Fig. 3.

Fig. 3

Plot of the chi-square distribution. The vertical axis is the density of the distribution, and the horizontal axis is the $a({\mathbf{z}}^{\prime })$ value. Degree of freedom $d=16$.

In Fig. 3, the graph shows the likelihood of the $a({\mathbf{z}}^{\prime })$ value of ${\mathbf{z}}^{\prime }$ sampled from the normal model following the standard Gaussian distribution. When an $a({\mathbf{z}}^{\prime })$ value is high, the probability of being a normal sample is low; therefore, the sample can be regarded as anomalous. Hence, it is possible to classify normal and anomalous samples by predetermining any upper probability on the graph and setting a one-dimensional threshold.

3.3.

Adversarial Autoencoder

Although images are high-dimensional data, they can be compressed to lower-dimensional features in the latent space. This is because normal samples are assumed to have common features. An autoencoder is a low-dimensional feature extractor for neural networks. An autoencoder is composed of two networks: encoder (En) and decoder (De). En is trained to extract features as latent vector $\mathbf{z}\sim q(\mathbf{z})$ from input $\mathbf{x}\sim p_{\text{data}}(\mathbf{x})$, where $p_{\text{data}}(\mathbf{x})$ is the data distribution of the input samples. De is trained to reconstruct the input $\mathbf{x}$ from $\mathbf{z}$. The loss function is shown as follows:

Eq. (4)

$$L_{AE}=E_{\mathbf{x}\sim p_{\text{data}}}[{(\mathbf{x}-\mathrm{De}(\mathrm{En}(\mathbf{x})))}^2].$$

Principal component analysis¹³ is another conventional dimensional reduction method, but it can only map linearly from the high-dimensional space to the low-dimensional latent space. The autoencoder enables nonlinear mapping using activation functions and deep layers. This leads to the model extracting more representative features of complex structured data because the projection functions En and De are more flexible.

Although low-dimensional features of input samples can be acquired by the autoencoder, the distribution of the features in the latent space cannot be specified. Therefore, to apply the Hotelling’s T-squared method described in Sec. 3.2 to the distribution of the features extracted by the autoencoder, we employ an AAE consisting of the autoencoder and discriminator networks shown in Fig. 4. The AAE allows for matching of the distribution of the latent space to an arbitrary distribution by an adversarial manner.⁷ To incorporate Hotelling’s T-squared method to the deep generative model, we train the AAE with an adversarial loss between the distribution of the encoded latent vectors and the standard Gaussian distribution. Furthermore, we assume the real-world situation in which a large number of normal samples and a small number of anomalous samples are available. The adversarial training with such imbalanced samples facilitates the normal samples being mapped to the high density of the standard Gaussian distribution and the anomalous samples being mapped to the low density. This means that the AAE constructs a normal model that follows the standard Gaussian distribution in the latent space. Therefore, it is possible to apply Hotelling’s T-squared method in the latent space. The reason for defining the arbitrary distribution as a standard Gaussian distribution is to simplify the anomaly score calculations described in Sec. 3.2.

Fig. 4

Architecture of an AAE consisting of an autoencoder and discriminator. In the autoencoder, the En extracts latent vector $\mathbf{z}$ from input images $\mathbf{x}$ sampled from $p_{\text{data}}(\mathbf{x})$, and the De reconstructs $\mathbf{x}$ from $\mathbf{z}$. The discriminator determines whether the input is sampled from standard Gaussian distribution $p(\mathbf{z})$ or latent distribution $q(\mathbf{z})$.

The discriminator is trained to determine whether the input vector is sampled from latent distribution $q(\mathbf{z})$ or from standard Gaussian distribution $p(\mathbf{z})$. In contrast, the En is trained to approximate $q(\mathbf{z})$ to $p(\mathbf{z})$. These actions are called adversarial training and are defined in a loss function as Eq. (5). The En is trained to minimize and the De is trained to maximize function $V$, and $E$ means cross entropy between a subscript and square brackets.

Eq. (5)

$$\underset{\mathrm{En}}{\min }\underset{\mathrm{D}}{\max }V(\mathrm{D},\mathrm{En})=E_{\mathbf{z}\sim p}[\log (\mathrm{D}(\mathbf{z}))]+E_{\mathbf{x}\sim p_{\text{data}}}[\log (1-\mathrm{D}(\mathrm{En}(\mathbf{x})))].$$

Discriminator $(\mathrm{D})$ updates its own parameters to output $\mathrm{D}(\mathbf{z})=1$ when input vector $\mathbf{z}$ is sampled from $p(\mathbf{z})$ and output $\mathrm{D}(\mathbf{z})=0$ when $\mathbf{z}$ is sampled from $q(\mathbf{z})$. Therefore, when the discriminator maximizes Eq. (5), it can determine whether the input $\mathbf{z}$ is sampled from $p(\mathbf{z})$ or $q(\mathbf{z})$. The loss function of the discriminator $L_{\mathrm{D}}$ is transformed from Eq. (5) to Eq. (6) as follows:

Eq. (6)

$$\begin{gathered} L_{\mathrm{D}}=\max E_{\mathbf{z}\sim p}[\log (\mathrm{D}(\mathbf{z}))]+E_{\mathbf{x}\sim p_{\mathrm{data}}}[\log (1-\mathrm{D}(\mathrm{En}(\mathbf{x})))] \\ =\max E_{\mathbf{z}\sim p}[\log (\mathrm{D}(\mathbf{z}))]+E_{\mathbf{z}\sim q}[\log (1-\mathrm{D}(\mathbf{z}))]. \end{gathered}$$

In contrast, Eq. (5) is minimized when the En can approximate $q(\mathbf{z})$ to $p(\mathbf{z})$ sufficiently, and the loss function of the En $L_{\mathrm{En}}$ can be transformed from Eq. (5) to Eq. (7) as follows:

Eq. (7)

$$\begin{gathered} L_{\mathrm{En}}=\min E_{\mathbf{x}\sim p_{\mathrm{data}}}[\log (1-\mathrm{D}(\mathrm{En}(\mathbf{x})))] \\ =\min E_{\mathbf{z}\sim q}[\log (1-\mathrm{D}(\mathbf{z}))]. \end{gathered}$$

To summarize, the AAE is trained to repeat the following procedure:

• 1. update En and De parameters to minimize Eq. (4),

• 2. update $D$ parameters to maximize Eq. (6), and

• 3. update En parameters to minimize Eq. (7).

4.

Experiments

We performed experiments with the proposed method using the AAE and x-ray CT images of solder joints on PCBs. The anomaly detection procedure of the proposed method is as follows:

The architecture used in the experiments is shown in Fig. 5. Each of the sliced images consisted of eight-layer images, as shown in Fig. 1. We resized the sliced images to $64\times 64\text{pixels}$ and input $\lambda =8$ sliced images to the AAE network as eight channels.

Fig. 5

Parameters of the AAE autoencoder. Numbers following words are filter sizes. “s2” means a convolution with a stride of 2. The input of the discriminator is from the autoencoder’s latent vector at dense layer 16 or the vector sampled from the standard Gaussian distribution.

We compared our method with a method using handcrafted features and OCSVM. This is to show that our method using features extracted automatically by AAE is superior to the classification by machine learning using features designed by human experts. The handcrafted features designed by human experts were four-dimensional features: the substrate area, head-in-pillow area, circularity, and luminance ratio.

The experimental results are shown in Table 1. In this table, the result for handcrafted features + OCSVM was with the condition of using all normal samples for training the OCSVM, and we show another result of training the OCSVM with fewer normal samples in Table 2. This result shows that the accuracy improved as the number of training samples increased; however, it was inferior to the proposed method even if all normal samples were used for training the OCSVM. The AAE architecture used in the experiments is shown in Fig. 5. The inputs to the network were $64\times 64\times 8$. We used the AAE parameters of batch size = 64 and epoch = 100, and the OCSVM parameter $\gamma =0.11$ and radial basis function kernel. Our code is available at https://github.com/rearwist3/aae_solder_tf. We chose 100 epochs empirically by observing the performance of the model every 20 epochs over 200 epochs. Figure 6(a) contains all of the results, and Fig. 6(b) omits the results at 20 epochs to show the details of the false positive rate (FPR) from 40 to 200 epochs. Because low FPR was obtained at 100 epochs and 120 epochs with 10 anomalous training samples, we chose 100 epochs. In the network, the computation time of the learning phase through 100 epochs was $\sim 80\min$ on an RTX 2080 Ti GPU.

Table 1

Comparison of our AAE method (ours) with handcrafted features + OCSVM.

Table 2

Comparison of results of handcrafted features + OCSVM with different number of samples used for training OCSVM. The rows denote the results for using each number of training samples. (small, medium, and large).

Fig. 6

FPR of every 20 epochs on 10 anomalous training samples is used in the training dataset.

We set the threshold as 100% true positive rate (TPR) in both models to avoid classifying anomalous samples as normal. FPR of handcrafted features + OCSVM was 1.10% after training with 3,510,000 normal samples. In contrast, AAE + Hotelling’s T-squared method could be trained with only 40,000 normal and 10 anomalous samples, and it could classify normal and anomalous samples with fewer false positives. To verify the results, we selected 10 anomalous training samples at random three times and trained the network with each dataset for 100 epochs. Mean and standard deviation of the resulting FPR are $0.07\pm 0.01\%$. We show the results when training the network with 0, 20, 50, and 100 anomalous samples to prove that including anomalous samples in the training dataset improves anomaly detection performance and to find the optimal balance between normal and anomalous samples for training the network. The resulting FPR is 5.15%, 0.93%, 0.10%, and 1.25%, respectively. The results confirm that including anomalous samples in the training dataset is effective in the anomaly detection method and the case with 10 anomalous training samples had the best performance and the fewest anomalous training samples.

Moreover, we compared our method with classification by a binary classifier, which is a typical anomaly detection method using deep learning. By this experiment, we show the effectiveness of the proposed method under the condition in which a sufficient number of anomalous samples for the training classifier cannot be guaranteed. The result when the binary classifier is trained with a large number of normal and a small number of anomalous samples is shown in Table 3. The classifier could not classify normal and anomalous samples when the number of normal and anomalous training samples was imbalanced. Moreover, we show the result for the binary classifier under the condition in which the number of normal and anomalous samples is not imbalanced in Table 3. In this experiment, we reduced the number of normal samples to match the number of anomalous samples to equalize each sample class and then trained the classifier. Neither result was as good as that of the proposed method, and we thus conclude that the proposed method is effective when the number of anomalous samples is small.

Table 3

The results when the samples were classified by the binary classifier. The first row denotes the results of training the classifier with the imbalanced dataset (without undersampling). The second row denotes the results of undersampling the dataset (with undersampling).

5.

Conclusion

In this paper, we propose a method for inspecting solder joints on PCBs by an anomaly detection method using an AAE. We captured sliced images of solder joints using x-ray CT, and the sliced image features following the standard Gaussian distribution were extracted by the AAE. Defects were detected by applying Hotelling’s T-squared method to these features. Experimental results showed that the AAE could classify normal and anomalous samples with few false positives even when the number of data samples was small. However, when compressing high-dimensional data to low-dimensional space, the number of low latent dimensions required for the full expression of high-dimensional data depends on the inputs, and we need to select the optimal number of latent dimensions for every dataset. Statistical implementation of methods to optimize the number of latent dimensions will be studied in future work.