Extreme learning machine for a new hybrid morphological/linear perceptron

Highlights

• New HMLP model having linear neurons and morphological units in hidden layer.

• A morphological unit (dendrite) computes the infimum of an erosion and anti-dilation.

• Circumventing non-differentiability issues, the HMLP can be trained using ELM.

• Higher mean classification rate than related models in some benchmark problems.

• Significant statistical difference compared to other models except the MP/CL which is very susceptible to the curse of dimensionality.

Abstract

Morphological neural networks (MNNs) can be characterized as a class of artificial neural networks that perform an operation of mathematical morphology at every node, possibly followed by the application of an activation function. Morphological perceptrons (MPs) and (gray-scale) morphological associative memories are among the most widely known MNN models. Since their neuronal aggregation functions are not differentiable, classical methods of non-linear optimization can in principle not be directly applied in order to train these networks. The same observation holds true for hybrid morphological/linear perceptrons and other related models. Circumventing these problems of non-differentiability, this paper introduces an extreme learning machine approach for training a hybrid morphological/linear perceptron, whose morphological components were drawn from previous MP models. We apply the resulting model to a number of well-known classification problems from the literature and compare the performance of our model with the ones of several related models, including some recent MNNs and hybrid morphological/linear neural networks.

Introduction

Along with morphological associative memories (MAMs), morphological perceptrons (MPs) were among the first morphological neural network (MNN) models that appeared in the literature (Ritter et al., 1999, Ritter and Sussner, 1996, Ritter and Sussner, 1997, Ritter et al., 1998, Sussner, 1998, Sussner and Esmi, 2011, Sussner and Valle, 2006). The commonality shared by all MNNs is that their nodes, called “morphological neurons”, apply a “morphological operator” before applying an activation function. Unfortunately, there is no precise definition of a morphological operator as far as applications in image processing and analysis are concerned. According to Henk Heijman’s seminal book entitled “Morphological Image Operators” (Heijmans, 1994): “Any attempt to find a formal definition of a morphological operator, however, would lead inevitably to the following dilemma: either it would be too restrictive, excluding operators that should not be excluded a priori, or it would be too general, leading to a ‘theory of everything’”.

However, when resorting to the algebraic framework of mathematical morphology (MM), one finds exact definitions of the four types of elementary operators of MM on complete lattices. These four operators are erosion, dilation, anti-erosion, and anti-dilation (Banon & Barrera, 1993). Banon and Barrera have shown that every mapping between complete lattices can be expressed both as a supremum of pair-wise infima of erosions and anti-dilations and as an infimum of pair-wise suprema of dilations and anti-erosions. In view of this fact, it is interesting to observe that the overall computations at every node in the output layer of an MP can be represented as a supremum of pair-wise infima of gray-scale erosions and anti-dilations (Sussner & Esmi, 2011). The level $0$ curve of an infimum of a gray-scale erosion and an anti-dilation represents a hyperbox. Several constructive (hyperbox) learning algorithms for training MPs (Sussner, 1998), in particular competitive training algorithms (Sussner and Esmi, 2009a, Sussner and Esmi, 2009b, Sussner and Esmi, 2011), have appeared in the literature. Other hyperbox learning algorithms were used by Ritter, Urcid, Sossa and Guevara to train parameters of dendritic MNNs (Ritter and Urcid, 2003, Sossa and Guevara, 2014). In classification problems, each resulting hyperbox encloses patterns, all or most of which belong to a single class. Note that hyperbox-based algorithms were also independently developed by Kaburlasos et al. for training fuzzy lattice neurocomputing models (Kaburlasos et al., 2007, Kaburlasos and Petridis, 2000).

One of the main reasons why researchers devised constructive training algorithms in order to train morphological perceptrons or dendritic MNNs, instead of resorting to standard methods of non-linear optimization, is the fact that the morphological operations used in these models are not differentiable. In their recent approaches towards training dendritic morphological and hybrid morphological/linear neural networks via stochastic gradient descent (Hernández et al., 2018, Zamora and Sossa, 2017), Zamora, Sossa, et al. apparently dealt with locations where the partial derivatives do not exist set by setting the search directions equal to $0$. In any case, the vector of search directions is not continuous, which is certainly a disadvantage. Also note that at locations where the function described by a dendrite is differentiable, all but one of its partial derivatives with respect to a weight are equal to $0$.

An alternative for training the weights of morphological neurons without encountering these problems consists in smoothing the morphological operations. This approach was employed by Pessoa and Maragos in their hybrid morphological/rank/linear neural network (MRL-NN) (Pessoa & Maragos, 2000) as well as in Araújo et al.’s morphological and hybrid morphological/linear neural networks (Araújo, 2012, Araújo et al., 2017, Araújo and Sussner, 2010). Finally, note that MRL-NNs, other hybrid models, and dendritic MNNs can also be trained by using methods of evolutionary computation (Araújo and Ferreira, 2013, Arce et al., 2018).

Our approach for training hybrid morphological/linear neural networks differs greatly from all aforementioned approaches since it is based on Huang et al.’s extreme learning machine. According to Huang, Zhu, and Siew (2006), training a network using extreme learning machine (ELM) (Huang et al., 2006) is computationally inexpensive compared to evolutionary optimization (Simon, 2013) and classical neural network training algorithms and generally leads to a good generalization performance without requiring some form of regularization in order to avoid “overfitting” (Bishop, 1995, Bishop, 2006, Ripley, 1996). The hybrid morphological/linear perceptron (HMLP) introduced in this paper is a feedforward neural network model that includes hidden morphological units taken from the previous MP models. The outputs of these morphological units and traditional semi-linear neurons are then combined using linear aggregation functions in the output neurons. Therefore, an ELM-based approach can effectively be employed in order to learn the weights of these linear combinations. In this paper, we apply our HMLP approach to a number of classification problems from the literature and compare the resulting classification rates with the ones of several related models.

The paper is organized as follows: Section 2 reviews some relevant concepts of lattice theory and mathematical morphology,including a few pertinent comments on gray-scale mathematical morphology. After discussing morphological perceptrons and their relationship with some related models from the literature, as well as providing some motivation, we introduce a new hybrid morphological/linear perceptron (HMLP) in Section 3. Then we present an extreme learning machine approach for HMLP training in Section 4 and compare the classification performance achieved by our model with related models from the recent literature in Section 5. We finish the paper with some concluding remarks.