2.1.

Hyperspectral Dataset

The limited availability of benchmarks datasets for the HSICD task is considered a major limitation to the RS community. In this work, we consider three binary HSICD datasets, namely the Bay Area, Santa Barbara,³¹ and multiclass Hermiston datasets,²⁶ as shown in Table 1. The availability of pixel-based annotated masks for each dataset enables analytical evaluation for their experimental results.

Table 1

Specification and data distribution of HSICD benchmarks.

2.1.1.

Bay Area dataset

This dataset consists of two coregistered hyperspectral images over the city of Patterson, California, of a section with (600 × 500) pixels captured by the AVIRIS sensor. Each image contains 224 spectral bands with a spatial resolution of about 30 m per pixel. The images were acquired in 2007 and 2015, respectively. The bitemporal images, as well as the ground truth, are shown in Fig. 1(a).

Fig. 1

Benchmark datasets: (a) Bay Area dataset, (b) Santa Barbara dataset, and (c) Hermiston dataset.

2.2.

Proposed Workflow

In this paper, we present an efficient workflow for HSICD, as shown in Fig. 2, which is composed of four main phases: preprocessing, training, testing, and evaluation. The proposed workflow was inspired by semantic segmentation due to their booming performance in several applications, such as scene comprehension,³² processing satellite images,^15,33 and object detection in satellite images.³⁴ UNet model,³⁵ which is considered a famous and effective semantic segmentation architecture, is used in the training phase to identify the change regions. In general, UNet employs the traditional encoder–decoder scheme. The input image is compressed into a dense feature vector by the encoder block. The spatial dimension of the feature vector is gradually reduced to obtain intense high discriminative representation. On the other hand, the feature vector has to spatially expand progressively to produce a segmented image. Several approaches such as bilinear interpolation and transposed convolution have been employed in the decoder block to match the original image dimensions.

Fig. 2

The proposed phases for segmentation-based HSICD (HSICD_workflow).

The proposed workflow aims to simulate real-life scenarios in which the imbalanced class problem is a major challenge, especially in satellite imageries. Finally, the performance of the proposed workflow was measured in terms of precision, recall, $F$-measure, Kappa-coefficient, and overall accuracy (OA). The proposed workflow includes the following four primary phases:

In this work, we employed UNet model and four of its variants in the proposed workflow of HSICD, namely traditional UNet,³⁵ residual UNet (R UNet), attention UNet (Att- UNet), recurrent residual UNet (R2 UNet), and attention recurrent residual UNet (Att-R2 UNet).Traditional UNet is shown in Fig. 3(a). The first variant of UNet is Residual UNet,³⁸ which was introduced as an extension to benefit from the residual learning as shown in Fig. 3(b). The second variant is attention UNet,³⁷ which incorporates attention gates (AGs) to produce soft region proposals to highlight salient region of interest (ROI) features and suppress feature activations from irrelevant backgrounds. AG was plugged after the standard convolutional block in the decoder. The AG architecture³⁷ is shown in Fig. 3(c). The third variant is the recurrent residual UNet architecture,³⁶ shown in Fig. 3(d), in which the recurrent convolutional operation is measured at a discrete time. The last variant incorporated residual, recurrent, and AGs³⁷ into each encoder and decoder block to enrich information flow and enforce a semantic discriminative intermediate feature map at every scale.

Fig. 3

Variant UNet architectures: (a) traditional UNet, after,³⁵ (b) residual UNet block,³⁶ (c) attention UNet block,³⁷ and (d) recurrent residual UNet block.

Typically, the loss functions applied in segmentation are categorized into distribution-based losses (minimize dissimilarity between two distributions) ad region-based losses (minimize the mismatch or maximize the overlap regions between the two images); details are given in Refs. 39,40. A common practice is to evaluate small subset of available loss functions to avoid the impracticability of experimenting on all available loss functions. In this work, we compared the performance of five widely used loss functions, namely cross-entropy loss, focal loss, Tversky loss, dice loss, and contrastive loss, to evaluate their performance in imbalanced HIS datasets.

3.1.

Evaluation Metrics

The proposed HSICD workflow performance was evaluated based on precision, recall, $F$-measure, kappa coefficient, and OA.

Precision computed by Eq. (1) indicates the average of images that are correctly identified to the total number of images that are correctly and noncorrectly identified with the reference input:

Recall, depicted in Eq. (2), is defined as the average number of images that are correctly identified out of the total number of images that are correctly and noncorrectly identified:

$F1$ score is defined by Eq. (3). If the obtained value reaches 1, it is classified as best, and if it reaches 0, as worst:

Kappa coefficient is calculated as

Finally, OA represents the proportion of correctly identified pixels to the total number of pixels.

3.2.

Experiment Setup

In all experiments, each dataset was separated into three subsets, namely, training (60%), testing (20%), and evaluation (20%). We implemented a 10-fold cross-validation strategy to ensure balanced outcomes; patches in training and testing subsets are nonoverlapped. In the training and evaluation phases, the mutually exclusive dataset ensures an event that does not split the training and testing datasets.

For all variant UNet models, we eliminated one layer from the original UNet architecture and implemented a three-layer ($\#L=3$) UNet version to cope with small input patches ($16\times 16\text{pixels}$) as shallower architectures are easier to train due to the relatively smaller number of hyperparameters to be optimized. The encoder is preceded by a bridge layer and a three-layer, skip-linked decoding path. The adaptive moment estimation (Adam)⁴¹ was selected to train the models due to its minimal tuning parameters requirement. The models were trained with a minibatch size of 16, and the number of epochs and the learning rate were set to 20 and 0.0001, respectively. These parameters were chosen based on their empirically adequate performance. We conducted all experiments using an Intel (R) Core i7 3.40 GHz CPU with NVIDIA GeForce GTX 1080-Ti. Due to the computing resources limitations, the optimization of thetraining algorithm parameters may further improve the performance.

3.3.

Results and Discussion

We conducted ample experiments to thoroughly analyze each UNet model’s performance and inference time. The Bay Area dataset obtained results are shown in Table 2, which compares the performance of the five implemented models (UNet, R-UNet, Att-UNet, R2-UNet, and Att-R2-UNet). The performance of the results was calculated from the test set results over the 10-fold cross validation. The implemented Att-R2-UNet architecture performed better on semantic segmentation with respect to several metrics, with the highest OA equals to 94.99% and a maximum precision score of 93.23%. The lowest OA score was reported for the traditional UNet model and is equal to 91.5%.

Table 2

Results obtained for HSICD using variant UNet models on the Bay Area dataset.

For the residual UNet and recurrent residual UNet architectures, the obtained OA results are auspicious (0.93 and 0.92), despite their naive architectures. Finally, the attention UNet and attention recurrent residual UNet architectures present higher performance in comparison with the traditional UNet and residual UNet models (0.93 and 0.95) since the spatial pyramid pooling outcome is combined with recurrent and recalibrated features from the encoder blocks. Overall, the obtained results revealed that the more naive decoder’s architecture leads to lower test accuracy.

Furthermore, the diversity and limitations of the performance of the proposed workflow can also be confirmed by the results in Table 3 obtained on the Santa Barbara dataset. It can be observed that there is a trade-off between the simplicity of network architecture and the obtained accuracy. More specifically, the UNet and residual UNet architectures present a relatively comparable accuracy. The same can be observed for the Att-UNet architecture; nonetheless, the OA was improved at the cost of the Cohen’s kappa metric. Thus, the R2-UNet model and the Att-R2 UNet can be considered to be the most effective since the OA for both of them are very close. Overall, almost all UNet models denoted comparable accuracies. The Att-R2 UNet model achieved the best performance numerically and visually in both the Santa Barbara and Bay Area datasets.

Table 3

Results obtained for HSICD using variant UNet models on the Santa Barbara dataset.

Figures 4 and 5 show the visual results of the obtained change maps from variant UNet segmentation models. The residual UNet model presents adequate performance for both benchmark datasets. On the contrary, the traditional UNet model demonstrated the lowest accuracy in identifying positive and negative changes. Moreover, the traditional UNet model fails to generate a change map that correctly captures change and no-change regions. The accuracy is significantly improved based on the visual results by integrating recurrent and residual learning. Furthermore, the rich salient regions obtained from AGs in the Att-UNet and R2 UNet models tend to be more robust in binary change identification; however, some false positives were reported.

Fig. 4

Bay Area dataset change map: (a) ground-truth, (b) traditional UNet, (c) R-UNet, (d) Att- UNet, (e) R2-UNet, and (f) Att-R2-UNet.

Fig. 5

Santa Barbara dataset change map (a) ground-truth, (b) traditional UNet, (c) R UNet, (d) Att- UNet, (e) R2 UNet, and (f) Att-R2 UNet.

Next, we evaluated the efficiency of the proposed workflow at detecting multiclass changes using the Hermiston dataset as illustrated in Table 4. In particular, the obtained OA for traditional UNet demonstrated the worst value 0.94 (OA). However, the utilization of residual and recurrent blocks enhanced the accuracy of R-UNet (0.989) and R2-UNet (0.953). Visually, Figs. 6(a)–6(c) demonstrated the enhancement of change map when incorporating residual and recurrent blocks. Moreover, attention mechanism shows more robust results, especially in small ROIs. Att-R2 UNet achieved the highest OA (0.991) on the pixels compared with all other UNet architectures. The obtained results showed that all UNet models could learn effectively change features from hyperspectral images in multiclass change cases. To sum up, all CD experiments confirmed that the integration of residual, recurrent, and attention mechanism facilitates a spectral–spatial–temporal change feature to be constructed effectively.

Table 4

Results obtained for HSICD using variant UNet models on the Hermiston dataset.

Fig. 6

Hermiston dataset change map (a) ground-truth, (b) traditional UNet, (c) R-UNet, (d) Att-UNet, (e) R2-UNet, and (f) Att-R2-UNet.

Furthermore, we carried out comparison between the deployed models to analyze the execution time and memory required for inference. The average inference time and the number of parameters of each model are given in Table 5. Overall, traditional UNet among all variant models presents the fastest in terms of inference time. The R-UNet and R2-UNet models demonstrate a higher inference time in spite of the number of parameters being lower compared with the traditional UNet model. This is justifiable because of the residual operations employed in both models at the encoding and decoding stages. The attention residual recurrent model presents the highest number of parameters allocation and displays the best performance for both binary and multichange identification cases. In conclusion, the recurrent residual UNet model was an ideal solution for binary and multichange identification for the hyperspectral problem with its high performance and relatively comparable inference speed.

Table 5

Comparison of variant UNet models in terms of mean inference time and number of parameters.

Next, we conducted various experiments to evaluate the following loss functions: focal loss, Dice loss, Tversky loss, and contrastive loss on the three HSI benchmarks using the standard UNet architecture described above. Based on the results in Table 6 from the Bay Area dataset, we select the contrastive loss, Dice loss, and focal loss as the top three performing loss functions. As shown in Table 6, in Hermiston dataset, the focal loss was associated with the best recall–precision balance, and it outperformed the contrastive loss and dice loss in recall and precision scores.

Table 6

UNet performance using variant loss functions on hyperspectral datasets. Numbers in boldface denote the highest values for each metric.

Finally, Figs. 7, and 8 present Bland–Altman plots and linear regression plots for area (segmented) and area (truth) for the Bay Area dataset and Santa Barbara dataset, respectively. This experiment was conduct using the standard UNet to visualize the robustness of the proposed workflow to identify the change and no-change zones. Specifically, the linear regression analysis (Figs. 7 and 8) indicates correlation with $R2=0.255$ and 0.358 for the identification of change zones for the Bay Area and Santa Barbara datasets, respectively. On the other hand, $R2=0.441$ and 0.435 for the unchanged zones. Bland–Altman plots indicate a slight bias for detecting change zones detection. Figure 9 shows Bland–Altman plots and linear regression plots for each class in the Hermiston dataset.

Fig. 7

(a) Linear regression results and (b) Bland–Altman plots for the comparison of change and no-change areas detected by the proposed workflow and corresponding ground truth for the Bay Area dataset.

Fig. 8

(a) Linear regression results and (b) Bland–Altman plots for the comparison of change and no-change areas detected by the proposed workflow and corresponding ground truth for the Santa Barbara dataset.

Fig. 9

(a) Linear regression results and (b) Bland–Altman plots for the comparison of five change class detected by the proposed workflow and corresponding ground truth for the Hermiston dataset.