Abstract
This paper proposes a perceptual medical image fusion framework based on morphological component analysis combining convolutional sparsity and pulse-coupled neural network, which is called MCA-CS-PCNN for short. Source images are first decomposed into cartoon components and texture components by morphological component analysis, and a convolutional sparse representation of cartoon layers and texture layers is produced by prelearned dictionaries. Then, convolutional sparsity is used as a stimulus to motivate the PCNN for dealing with cartoon layers and texture layers. Finally, the medical fused image is computed via combining fused cartoon layers and texture layers. Experimental results verify that the MCA-CS-PCNN model is superior to the state-of-the-art fusion strategy.
1. Introduction
In clinical applications, medical images include anatomical images and functional images. Anatomical images provide information of dense structures [1], for instance, X-ray computed tomography (CT) and magnetic resonance imaging (MRI). Functional images reflect information of blood flow and blood activity [2], for instance, positron emission CT (PET) and single-photon emission CT (SPECT). Medical images with single modality do not provide sufficient information in diagnosing diseases; medical image fusion (MIF) technology provides an effective method via merging medical images with different modalities into a comprehensive MIF image to aid radiologists for better diagnosis [3–5].
Many MIF algorithms have been addressed in the last dozen years. These methods include the multiscale decomposition- (MSD-) based fusion strategy [6–10], sparse representation- (SR-) based fusion strategy [11], and pulse-coupled neural network- (PCNN-) based fusion strategy [12, 13]. To pursue satisfactory fusion performance, attempts were made to use the PCNN based on MST [14–16]. PCNN is a cat visual cortex biologically inspired neural network, which is used in medical image fusion. Huang et al. [17] integrated non-subsampled contourlet transform (NSCT) with the PCNN for SPECT and CT image fusion. Non-subsampled shearlet transform (NSST) was combined with the PCNN to fuse medical images [18]. However, NSCT- or NSST-based fusion strategy has high computational complexity due to proper contours, which may limit the fusion performance. Furthermore, normalized coefficient values are employed to stimulate the PCNN, which may cause detail loss and blurring effect in the fused image. Electrophysiological experiments have proved that the neuron representations of complex stimulation in the cat visual cortex are represented by sparse coding [19–21]. Morphological component analysis (MCA) has been widely studied as effective image decomposition. Combining MCA with SR can acquire the SR of cartoon and texture components of an image [22, 23]. To resolve the disadvantage produced by patch coding, convolutional sparse representation (CSR) has been shown to be more effective than sparse representation in extracting features [24]. It is implemented on the whole image instead of a local image patch. Based on the above considerations, this paper presents a medical image fusion algorithm using convolutional sparsity to stimulate the PCNN based on morphological component analysis (MCA-CS-PCNN). Source images are first decomposed into cartoon components and texture components by MCA, and CSR of cartoon layers and texture layers is obtained by prelearned dictionaries. Then, convolutional sparsity is employed to stimulate the PCNN for processing cartoon layers and texture layers. The MIF image is computed via combining fused cartoon layers and texture layers. We test the performance of the proposed MCA-CS-PCNN fusion method, and the experimental results verify the advantages of our fusion strategy.
2. Related Work
2.1. Convolutional Sparsity Based on Morphological Component Analysis (CSMCA)
Convolutional sparsity is a sparse representation model applying the convolutional form [24], which is based on an entire image rather than overlapped patch. The CSR is defined aswhere denotes an image, and denote the global sparse coefficient maps and dictionary filter, respectively, represents the convolution operator, and is the regularization parameter.
Morphological component analysis of an image is regarded as a linear combination of different components, which is defined as [23]where and denote cartoon components and texture components, respectively. According to CSR theory, the model of convolutional sparsity based on morphological component analysis (CSMCA) is expressed aswhere and denote the dictionary and convolution sparse coefficient corresponding to , respectively. and represent the dictionary and convolution sparse coefficient corresponding to , respectively. The image is computed and denoted by
2.2. Pulse-Coupled Neural Network
The diagrammatic diagram of the simplified PCNN is shown in Figure 1. There are three modules in the simplified PCNN model [12], which include the dendritic, linking modulation, and pulse generator, where feeding and linking input are built into the dendritic, denoted by and . and denote the linking modulation and the pulse generator, respectively. The simplified PCNN model is denoted bywhere , denote pixel locations, , represent the dislocation in the symmetric neighborhood around a pixel, and denote the synaptic weight matrices and the external stimulus, respectively, and are normalizing constants, and varies the weight of the linking field, which denotes the linking parameter. The threshold magnitude coefficient and attenuation coefficient are represented by and , respectively.
3. Proposed MIF Fusion Framework
3.1. MCA-CS-PCNN
The flowchart of the MCA-CS-PCNN framework is shown in Figure 2. Images A and B denote different source images, which are decomposed into cartoon components and the texture components by applying MCA, respectively. According to equations (1)–(4), the CSR of cartoon components and texture components is computed aswhere represents functions, described in Section 2.1, and denote the convolution sparse coefficient map of , respectively, and and denote the convolution sparse coefficient map of .
Next, the convolutional sparse representation is used to stimulate the PCNN because complex stimulation in the cat visual cortex is based on sparse coding. and are employed to stimulate the PCNN for processing and , respectively.where denotes the PCNN functions; the firing time matrices of and of are obtained according to equations (5)–(9), until the iteration number , where denotes the max iteration times, and then iteration stops.
Then, fused coefficients of the convolution sparse coefficient map in cartoon components are computed by
The fused coefficients of the convolution sparse coefficient map in texture components are computed by
The fused image of cartoon component and fused image of texture component are computed and denoted bywhere and are dictionaries.
Finally, the medical fused image is acquired and denoted as
3.2. Extension to Anatomical and Functional Image Fusion Based on MCA-CS-PCNN
The proposed MCA-CS-PCNN is extended to conduct anatomical and functional image fusion. Considering that functional images are pseudo-color images, the YUV color space transform has shown to be effective in processing pseudo-color images [10, 16]. Specifically speaking, a functional image with RGB is firstly transformed into the Y channel, U channel, and V channel. Then, the new Y channel is produced via the fusion of the Y channel and grayscale image based on MCA-CS-PCNN, and the new YUV is acquired via merging the new Y, U, and V. Finally, YUV is converted into RGB, and the medical fused image with color is obtained. The flowchart of the anatomical and functional image fusion strategy based on MCA-CS-PCNN is shown in Figure 3.
4. Experiments
4.1. Experimental Settings
To test and verify the performance of the MCA-CS-PCNN fusion algorithm, ten pairs of medical images with the same size of 256 × 256 pixels are used to conduct the experiments, including five pairs of anatomical image and functional image fusion and five pairs of anatomical image and anatomical image fusion (Figures 4 and 5). Five representative medical image fusion algorithms are selected for experimental comparison; they are convolutional sparse representation (CSR) [24], NSCT-based modified spatial frequency and PCNN (NSCT-MSF-PCNN) [14], guided filtering (GFF) [25], cross-scale coefficient selection (CSCS) [26], and sparse representation based on the Laplacian pyramid (LP-SR) [11]. Objective quality evaluation is important for image quality [27–31]. The existing fusion quality metrics include the human perception quality metric [32], feature mutual information quality metric [33], spatial frequency quality metric [34], standard deviation quality metric [11], nonlinear correlation information entropy metric [35], and mutual information metric [36]. In the above quality metrics, the higher the values of , , , , , and , the higher the fusion performance.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
4.2. Analysis of Experimental Results
In the example of anatomical and anatomical image fusion, we can see that the anatomical information of the bones or soft tissues is contained in the fused images by the six algorithms; still, differences between fused images can be clearly distinguished, such as focal regions blur (Figures 6(a), 6(c), and 6(d)), information of soft tissues regions are missing (Figures 6(b) and 6(e)). Our method obtains better performance than other methods. The example of anatomical and functional image fusion shows that the fused images obtained by GFF and CSCS lead to the loss of color information (Figures 7(a), 7(c), and 7(d)), and the NSCT-MSF-PCNN and LP-SR algorithms lead to poor visual effect, for instance, the details of the anatomical image are lost (Figures 7(b) and 7(e)). From the comparisons, our proposed algorithm demonstrates more advantages than the existing algorithms.
(a)
(b)
(c)
(d)
(e)
(f)
(a)
(b)
(c)
(d)
(e)
(f)
Tables 1 and 2 give the objective evaluation results of the proposed MCA-CS-PCNN fusion algorithm and five fusion methods via using objective fusion quality metrics. We mark best results employing the boldface in each row. Table 1 shows an objective evaluation of the fused image about anatomical image and functional image. We values are only slightly lower than LP-SR in the second pair of images of Figure 4. Our method achieves the significant superiority. From Table 2, it can be see that values are only slightly lower than GFF in the second pair of images of Figure 5, and the values of and in our proposed algorithm demonstrate advantages.
5. Conclusion
This paper proposes a perceptual medical image fusion framework based on morphological component analysis combining convolutional sparsity and pulse-coupled neural network, which is called MCA-CS-PCNN for short. It is basically based on the visual system feature that the cat visual cortex can produce complex stimulation, and the neuron representations of complex stimulation can be represented using sparse coding. To this end, we first decomposed source images into cartoon components and texture components by morphological component analysis, and convolutional sparse representation of cartoon layers and texture layers is obtained by prelearned dictionaries. Then, convolutional sparsity is employed to stimulate the PCNN for processing cartoon layers and texture layers. Finally, the medical fused image is computed via combining fused cartoon layers and texture layers. The experimental results verify that the proposed model can produce high performance, which is superior to the state-of-the-art fusion strategy.
Data Availability
The data used to support the findings of this study can be downloaded from http://www.med.harvard.edu/AANLIB/home.html.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (no. 82001912), Project funded by the China Postdoctoral Science Foundation (2018M642325), and Xu Zhou Science and Technology Program, China (KC19146).