Knowledge-aware deep framework for collaborative skin lesion segmentation and melanoma recognition

Highlights

• We propose a novel end-to-end deep framework that is able to perform skin lesion segmentation and melanoma recognition jointly, where the clinical knowledge is exploited and transferred with the mutual guidance between these two tasks.

• We design a lesion-based pooling and shape extraction module to transfer the lesion structure information from the skin lesion segmentation task to the melanoma recognition task, which assists the network to learn more informative feature representation for melanoma recognition.

• We propose a diagnosis guided feature fusion scheme to pass the lesion class information from the melanoma recognition task into the skin lesion segmentation task, which generates discriminative representations for different types of skin lesions.

• We design a recursive mutual learning method that further enhances the joint learning ability of the proposed model for both skin lesion segmentation and melanoma recognition.

Abstract

Deep learning techniques have shown their superior performance in dermatologist clinical inspection. Nevertheless, melanoma diagnosis is still a challenging task due to the difficulty of incorporating the useful dermatologist clinical knowledge into the learning process. In this paper, we propose a novel knowledge-aware deep framework that incorporates some clinical knowledge into collaborative learning of two important melanoma diagnosis tasks, i.e., skin lesion segmentation and melanoma recognition. Specifically, to exploit the knowledge of morphological expressions of the lesion region and also the periphery region for melanoma identification, a lesion-based pooling and shape extraction (LPSE) scheme is designed, which transfers the structure information obtained from skin lesion segmentation into melanoma recognition. Meanwhile, to pass the skin lesion diagnosis knowledge from melanoma recognition to skin lesion segmentation, an effective diagnosis guided feature fusion (DGFF) strategy is designed. Moreover, we propose a recursive mutual learning mechanism that further promotes the inter-task cooperation, and thus iteratively improves the joint learning capability of the model for both skin lesion segmentation and melanoma recognition. Experimental results on two publicly available skin lesion datasets show the effectiveness of the proposed method for melanoma analysis.

Introduction

Melanoma is one of the most malignant skin cancer that increases rapidly throughout the world [1], [2], [3], [4]. Timely treatment of the melanoma can efficiently improve the survive rate of the patients. Dermoscopic images captured by digital imaging devices, offer the magnified visualization of the melanoma, and thus assist the dermatologists in examining the melanoma based on a set of complex visual characteristic of the lesion. Computer-aided diagnosis (CAD) system provides an effective way that allows dermatologists’ clinical inspection of the skin lesion in dermoscopic images. A CAD system for melanoma analysis generally contains two crucial functions: lesion segmentation and melanoma recognition. Specifically, the task of segmentation aims to divide a dermoscopic image into the skin lesion parts and background parts, i.e., it is a pixel-wise classification process to generate more conceptual saliency information for melanoma analysis [5], [6], [7], [8], [9]. Meanwhile, melanoma recognition is an image-level classification task that aims to identify the skin lesion types, such as melanoma, seborrheic keratosis, and benign nevi [10], [11], [12], [13], [14].

Recent deep learning-based methods have shown their promising achievements in both lesion segmentation [15], [16], [17], [18], [19] and melanoma recognition [13], [14], [20], [21], [22], [23], [24], [25], [26], [27]. Existing approaches usually train the task-specific models to perform skin lesion segmentation and melanoma recognition separately, and do not explicitly consider the dermoscopists’ clinical criteria for melanoma inspection. For example, most of the existing methods [13], [14], [20], [21], [22], [23], [24], [25], [26], [27] heavily depend on the abstract feature representation obtained at the high layer of the network, and they often leverage the global information of the whole feature maps for melanoma identification, i.e., melanoma recognition is often conducted in a similar fashion as other general object classification tasks [28], [29]. However, in the domain of melanoma inspection, such a fashion has a high potential of losing important pathological patterns and morphology knowledge that are crucial for melanoma recognition. Generally, for dermatologists, they identify melanoma via certain prior criteria, like the statistical information of the lesion color’s variation and the border’s irregularity [25], [30], [31]. Moreover, clinical studies [11], [32], [33] have confirmed that the characteristics of the peripheral lesion are very useful morphological expressions for identifying melanoma, as shown in Fig. 1. However, these types of knowledge have not been explicitly considered yet by current deep learning-based methods [21], [22], [23], [24], [25], [26], [27].

In this paper, we propose a novel knowledge-aware framework that is able to exploit the clinical knowledge within the deep feature learning process for melanoma analysis. Specifically, in our framework, both the skin lesion segmentation task and the melanoma recognition task are learned and performed via a joint deep network, such that the clinical knowledge can be exploited and transferred with the mutual guidance and assistance of these two tasks.

Concretely, to promote the performance of the deep learning framework on melanoma recognition, we propose to incorporate the clinical knowledge by explicitly considering the morphological expression of the lesion area and also the periphery region. To achieve this, we design a novel lesion-based pooling and shape extraction (LPSE) module that transfers the lesion structure information obtained from the skin lesion segmentation task to the melanoma recognition task, and thus we are able to embed the morphological operation into our deep network. With the integration of morphological analysis of the skin lesion structure, our network is thus able to selectively learn the informative features containing useful statistical information from both the lesion center region and also the border region. Compared with the features produced by direct global average pooling in most of the existing deep learning-based methods [20], [21], [22], [23], [24], [25], [26], our network generates more discriminative lesion representation for melanoma recognition.

Melanoma and non-melanoma lesions generally have very different pathological feature representations (e.g., more border irregularity and inhomogeneous textures for melanoma lesions). With lesion class information, the segmentation network can generate more discriminative feature representation for detecting different types of skin lesions from dermoscopic images. To improve skin lesion segmentation performance for both the melanoma and non-melanoma classes, we propose a new unit called diagnosis guided feature fusion (DGFF) that incorporates the lesion diagnosis information from melanoma recognition task into skin lesion segmentation task. With the guidance of skin lesion class information learned from the melanoma recognition task, our DGFF achieves more discriminative feature representation for each lesion class and thus enhances the capability of our network in segmenting the skin lesion regions from the dermoscopic images.

Moreover, to seamlessly achieve both the aforementioned skin lesion segmentation task and the melanoma recognition task within our end-to-end network, we also propose a recursive mutual learning scheme that enables effective inter-task cooperation and recurrently improves the joint leaning capability of the model on both tasks. The proposed recursive mutual learning scheme systematically exploits the mutual guidance signals generated between skin lesion segmentation and melanoma recognition, and thus simultaneously boosts the performance of both tasks. The main contributions of this paper are summarized as follows:

$•$ We propose a novel end-to-end deep framework that is able to perform skin lesion segmentation and melanoma recognition jointly, where the clinical knowledge is exploited and transferred with the mutual guidance between these two tasks.

$•$ We design a lesion-based pooling and shape extraction module to transfer the lesion structure information from the skin lesion segmentation task to the melanoma recognition task, by embedding the morphological analysis into the feature learning process of our network, which helps generate informative clinically interested features for melanoma recognition.

$•$ We propose a diagnosis guided feature fusion scheme to pass the lesion class information from the melanoma recognition task into the skin lesion segmentation task, which generates discriminative representations for different types of skin lesions.

$•$ We design a recursive mutual learning method that further enhances the joint learning ability of the proposed model for both skin lesion segmentation and melanoma recognition.