Res-CR-Net, a residual network with a novel architecture optimized for the semantic segmentation of microscopy images

2.1.1. Transmission electron microscopy (EM) of rat liver cells.

2.1.2. Fluorescence microscopy (FM) of human skeletal muscle tissue sections.

2.2.1. Electron microscopy.

Grayscale images (262 × 400 px, reduced from the original 2622 × 4000 px) of rat liver cells were obtained as described in [14]. The segmentation task was to recognize nuclei, mitochondria, and anything else that is neither nuclei nor mitochondria (figures 1(a)–(d)).

Zoom In Zoom Out Reset image size

In this case, both images and ground truth binary masks were of dimensions (262 × 400 × 1), with three masks (one for each class) per image. Res-CR-Net was trained for 90 epochs with 8 images/8 × 3 masks (corresponding to one batch), and validated against four images/4 × 3 masks.

2.2.2. Fluorescence microscopy.

2.3.1. Image annotation and ground truth labels.

2.3.2. Data augmentation.

A flowchart of the architecture of Res-CR-Net is shown in figure 2. It combines two types of residual blocks, a Convolutional Residual Block (CONV RES BLOCK) and a Long Short Term Memory Residual Block (LSTM RES BLOCK), which are repeated in a linear path along which the dimensions of the intermediate feature maps remain identical to those of the input image and of the output mask(s).

• 1.  
  CONV RES BLOCK. The residual path of this block consists of three parallel branches of separable/atrous convolutions that produce feature maps with the same spatial dimensions as the original image. The kernel sizes and dilation rates used in this block can be customized at will. In this study we have used kernel sizes of [3,3], [5,5], and [7,7], with dilation rates of [1,1], [3,3], and [5,5], respectively.

Zoom In Zoom Out Reset image size

As stressed in [4], the rationale for using multiple kernel sizes and dilations is to extract object features at various receptive field scales. Res-CR-Net offers the option of concatenating (the default) or adding the parallel branches inside the residual block before adding them to the shortcut connection. A Spatial Dropout layer follows each residual block. A single STEM block, which differs from the CONV RES BLOCK only for lacking the initial Batch Normalization and Leaky ReLU operations (marked with a star symbol in figure 2), processes the initial input. n-1 CONV RES BLOCKS can be concatenated. Since the spatial dimensions of the output feature maps remain constant in the chain of repeated blocks, a dense version [15, 16] of the module (which is offered as an option in our deposited code) can be easily constructed without increasing the number of parameters by adding the output of each block to the output of all subsequent blocks.

• 2.  
  LSTM RES BLOCK. This block features a residual path with two orthogonal bidirectional 2D convolutional long short term memory (LSTM) [17] layers processing, respectively, the rows and columns of the input map from previous layers. For this purpose, the 4D tensor emerging from the previous layer first undergoes a virtual dimension expansion to a 5D tensor (i.e. from [4, 260, 400,3], [batch size, rows, columns, number of classes] to [4, 260, 400,3,1], [batch size, rows, columns, number of classes, number of channels]). In this example, the 2D LSTM layer accepts 260 consecutive tensor slices of dimensions [400,3,1] as the input data at each iteration. Each slice is convolved with a filter of kernel size [3,3,number of channels] with 'same' padding, and returns a slice of the same dimension. If more than one filter is used, the 3rd dimension of the resulting slice reflects the number of filters. For example, in one-directional mode the LSTM layer returns a tensor of dimensions [4, 260, 400,3,1 × number of filters]. In bidirectional mode it returns a tensor of dimensions [4, 260, 400,3,2 × number of filters]. In most applications a single filter suffices, but slightly more accurate results can be obtained by increasing the number of filters. The intuition behind using a convolutional LSTM layer for this operation lies in the fact that adjacent image rows share most features, which can be memorized in the LSTM unit. This type of intuition is perhaps somewhat related to the idea of representing the mean-field iterations of a CRF as a recurrent neural network [11]. Since the same intuition applies also to the image columns, in our example the expanded feature map of dimensions [4, 260, 400,3,1] from the earlier part of the network is transposed in the 2nd and 3rd dimension to a tensor of dimensions [4, 400, 260,3,1]. In this case the LSTM layer processes 400 consecutive tensor slices of dimensions [260,3,1] as the input data at each iteration, returning a tensor of dimensions [4, 400, 260,3,2 × number of filters], which is transposed again to [4, 260, 400,3,2 × number of filters]. The two LSTM output tensors are then added and the final dimension (whose size also depends on the number of filters used) is collapsed by summing its elements, leading to a final tensor of dimensions [4, 260, 400,3], which is added to the shortcut path. m LSTM RES BLOCKS can be concatenated. While using very few parameters, LSTM blocks are computationally expensive, and thus Res-CR-Net allows for not including them in the network architecture, if in a particular case they do not improve the segmentation accuracy.

A Leaky ReLU activation is used throughout Res-CR-Net. After the last residual block a softmax activation layer is used to project the feature map into the desired segmentation. The Dice coefficient, $D$ [18–22], and the Dice loss, ${L_D}$, defined as:

$$\begin{equation*}{L_D}\left( {\boldsymbol{\widehat y,y}} \right) = 1 - D = 1 - \frac{{2\mathop \sum \nolimits_i {{\hat y}_i}{y_i} + s}}{{\mathop \sum \nolimits_i {{\hat y}_i} + \mathop \sum \nolimits_i {y_i} + s}}\end{equation*}$$

with ${\boldsymbol{\widehat y} }\equiv \{ {\hat y_i}\}$, ${\boldsymbol{\hat y}_i} \in \left[ {0,1} \right]$ being the probabilities for the i-th pixel, $y \equiv \{ {y_i}\}$, ${y_i} \in \left\{ {0,1} \right\}$ being the corresponding ground truth labels, and $s$ a smoothing scalar, have been shown to increase performance over the cross entropy loss for semantic segmentation tasks [23], and thus are a popular choice among practitioners. In [8] it was found empirically that loss functions of the Dice class containing squares in the denominator behave better in pointing to the ground truth, and that faster training convergence can be obtained by complementing the loss with a dual form that measures the overlap area of the complement of the regions of interest. These observations, and additional theoretical considerations [8] led to the implementation of the Tanimoto loss, defined as:

$$\begin{equation*}{L_{\tilde T}}\left( {\boldsymbol{\widehat y},y} \right) = 1 - \tilde T\left( {\boldsymbol{\widehat y},y} \right)\end{equation*}$$

where $\mathop T\limits^\sim \left( {\boldsymbol{\widehat y},y} \right)$ is the Tanimoto coefficient with complement:

$$\begin{equation*}\mathop T\limits^\sim \left( {\hat y,y} \right) = \,\frac{{T\left( {\hat y,y} \right) + T\left( {1 - \hat y,1 - y} \right)}}{2}\end{equation*}$$

with $T\left( {\boldsymbol{\widehat y},y} \right)$ defined as:

$$\begin{equation*}T\left( {\boldsymbol{\widehat y},y} \right) = \frac{{\mathop \sum \nolimits_i {{\hat y}_i}{y_i} + s}}{{\mathop \sum \nolimits_i \left( {\hat y_i^2 + y_i^2} \right) - \mathop \sum \nolimits_i {{\hat y}_i}{y_i} + s}}\end{equation*}$$

While in this study we have used the Tanimoto loss function throughout, Res-CR-Net can use a variety of other losses, including categorical crossentropy, dice loss [18–22], categorical crossentropy + dice loss, Tanimoto loss without complement, categorical crossentropy + Tanimoto loss, all appropriately weighted to account for class imbalance. Weights are derived either with a contour aware scheme, by replacing a step-shaped cutoff at the edges of the mask foreground with a raised border that separates touching objects of the same or different classes [6], or with the inverse volume scheme [21], ${w_J} = V_J^{ - 2}$, where ${w_J}$ are the weights and ${V_J}$ is the total sum of true positives per class J, or with both. Regardless of the loss function used for training, in this study we used the unweighted Dice coefficient defined above as the primary metric to evaluate segmentation accuracy.

Res-CR-Net was implemented using Keras [24, 25] deep learning library running on top of TensorFlow 2.1 [26], and is publicly available at https://github.com/dgattiwsu/Res-CR-Net. Training and testing were conducted on the Wayne State University high performance computing grid equipped with Intel CPUs and Tesla V100-SXM2-16GB GPUs. Res-CR-Net can process images of any size, without constrains imposed by the down-pooling and up-sampling operations of an encoder-decoder architecture. Three types of masks can be used: (1) a single 3 or 4 channels binary mask/image is suitable to process up to 4 segmentations classes. (2) a single 1 channel (grayscale) mask/image can be used to process an arbitrary number of classes. In this case, the mask must be thresholded with different gray levels (i.e. a mask with three classes would have the regions corresponding to the three categories thresholded at 0,128,255 values). This type of mask is first converted to sparse categorical with each gray level corresponding to a different index (i.e. [0, 128, 255] goes to [0, 1, 2]). Then, pixels identified by indices are converted to one-hot vectors. (3) multiple 1 channel binary masks per image, with a separate mask for each class can also be used to process an arbitrary number of classes. In this case, the masks must be placed in different directories corresponding to the different classes.

When trained for 90 epochs on the EM dataset of 8 grayscale images (262 × 400 px) with 15-fold augmentation (see above), Res-CR-Net (configured with six CONV RES BLOCKS and two LSTM RES BLOCKS) achieved ∼90% segmentation accuracy (Dice coefficient: 92.2%, table 1) on the four images of the validation set (figure 3).

Zoom In Zoom Out Reset image size

The segmentation task in these images was to identify the regions occupied by the cell nuclei, the cytoplasm, and the mitochondria. Thus, three binary segmentation masks were associated with each grayscale image, and the network trained in each epoch on 120 different augmented images and corresponding 120 × 3 augmented binary masks as labels. A block diagram of the network architecture used for this task is shown in figure S1 (available online at stacks.iop.org/MLST/1/045004/mmedia). The total number of parameters refined was 58 978, the memory use was 9.79 GB/(batch of eight images).

It is worth noting that segmentation of mitochondria in electron micrographs from focused ion beam scanning electron microscopy (FIB-SEM) or automated tape-collecting ultramicrotome scanning electron microscopy (ATUM-SEM) has been the focus of multiple reports [27–31] in recent years. However, due to the variety of mitochondrial structures, as well as the presence of noise, artifacts, and other sub-cellular structures, mitochondria segmentation in EM has proven to be a difficult and challenging task. As apparent from figure 3, Res-CR-Net performs very well in this task, despite the small number of images used for training.

When trained for 90 epochs on the FM dataset of 10 images with 15-fold augmentation (see above), Res-CR-Net (configured with six CONV RES BLOCKS and one LSTM RES BLOCK) showed excellent refinement properties as attested by its convergence speed (∼30 epochs) and stability (up to 90 epochs) with respect to both the training and validation set (figure 4).

Zoom In Zoom Out Reset image size

The segmentation task in these images was to identify the cell nuclei, the cytoplasm, the boundaries between muscle cells, and regions that do not correspond to any of the previous three categories. Thus, four binary segmentation masks were associated with each color image, and the network trained in each epoch on 150 different augmented images and corresponding 150 × 4 augmented binary masks as labels. In this task Res-CR-Net also achieved ∼90% segmentation accuracy (Dice coefficient: 90.6%, table 1) on the six images of the validation set (figure 5). A block diagram of the network architecture used for this task is shown in figure S2, Supporting Information.

Res-CR-Net identified very accurately the positions and shapes of nuclei (DAPI stained in blue in the leftmost column, and as black spots in the center and rightmost columns of figure 5), clearly distinguishing them from other bright spots in the cell boundary regions, which are due instead to high local concentration of myosin. To determine whether nuclei recognition was somehow facilitated by the presence of a DAPI stained channel in the RGB images used for training, we have retrained Res-CR-Net after first converting all color images to grayscale. In this case, while Res-CR-Net showed somewhat decreased per pixel accuracy on the validation set (Dice coefficient: 87.8%, see also table 1), it was still able to identify correctly all the regions of interest (figure 6).

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

The performance of Res-CR-Net with the EM and FM datasets is summarized in table 1 with respect to the metrics (Dice coefficient, Precision, Recall, F1 score) most often used to evaluate neural networks performance.

However, prior to developing the intuition about the Res-CR-Net architecture we experimented with the traditional U-Net architecture. In fact, the LSTM RES BLOCK was initially implemented as the final block of a residual U-Net (for this reason called a Res-UR-Net ) with four levels in the encoding arm and four levels in the decoding arm, and tested with some of the color images in the FM dataset on a simpler segmentation task involving only three classes (figure 7).

Zoom In Zoom Out Reset image size

The residual blocks of Res-UR-Net were designed using the principles outlined in [7, 8], with two parallel convolutional branches of separable convolutions with kernel sizes [3,3], [5,5], respectively. A block diagram of Res-UR-Net for images of 512 × 512 px is provided in figure S3, Supporting Information. The depth of Res-UR-Net (four levels in both the encoder and decoder) is the same as that of the original U-Net [5, 6], slightly more than the three levels of Deep Res-U-Net [7], and slightly less than the six levels of Res-U-Net-a [8]. Thus, Res-UR-Net is in the middle of the range of depths that has been implemented in various flavors of U-Net style architectures. Several Res-U-Net architectures with or without a final LSTM block are also available for comparative testing in the Github deposition of Res-CR-Net (see above).

Using the complete FM dataset, we have now carried out a systematic comparison (table 2) of the Res-CR-Net used in this study against our own Res-UR-Net, trained under the exact same conditions (epoch number, batch size, augmentation parameters, loss, number of processors). From the results presented in table 2 it appears that Res-CR-Net achieves a level of segmentation accuracy that is at least as good if not slightly better than that achieved by Res-UR-Net, despite the latter using approximately 13 times more parameters. Just as noted in the case of Res-CR-Net, the training history of Res-UR-Net shows no significant overfitting (figure 8) of the training set vs. the validation set.

Zoom In Zoom Out Reset image size