A framework for comparing different image segmentation methods and its use in studying equivalences between level set and fuzzy connectedness frameworks

Abstract

In the current vast image segmentation literature, there seems to be considerable redundancy among algorithms, while there is a serious lack of methods that would allow their theoretical comparison to establish their similarity, equivalence, or distinctness. In this paper, we make an attempt to fill this gap. To accomplish this goal, we argue that: (1) every digital segmentation algorithm $\mathcal{A}$ should have a well defined continuous counterpart ${\mathcal{M}}_{\mathcal{A}}$, referred to as its model, which constitutes an asymptotic of $\mathcal{A}$ when image resolution goes to infinity; (2) the equality of two such models ${\mathcal{M}}_{\mathcal{A}}$ and ${\mathcal{M}}_{{\mathcal{A}}^{\prime }}$ establishes a theoretical (asymptotic) equivalence of their digital counterparts $\mathcal{A}$ and ${\mathcal{A}}^{\prime }$. Such a comparison is of full theoretical value only when, for each involved algorithm $\mathcal{A}$, its model ${\mathcal{M}}_{\mathcal{A}}$ is proved to be an asymptotic of $\mathcal{A}$. So far, such proofs do not appear anywhere in the literature, even in the case of algorithms introduced as digitizations of continuous models, like level set segmentation algorithms.

The main goal of this article is to explore a line of investigation for formally pairing the digital segmentation algorithms with their asymptotic models, justifying such relations with mathematical proofs, and using the results to compare the segmentation algorithms in this general theoretical framework. As a first step towards this general goal, we prove here that the gradient based thresholding model ${\mathcal{M}}_{▿}$ is the asymptotic for the fuzzy connectedness Udupa and Samarasekera segmentation algorithm used with gradient based affinity ${\mathcal{A}}_{▿}$. We also argue that, in a sense, ${\mathcal{M}}_{▿}$ is the asymptotic for the original front propagation level set algorithm of Malladi, Sethian, and Vemuri, thus establishing a theoretical equivalence between these two specific algorithms. Experimental evidence of this last equivalence is also provided.

Introduction

Most scientific contributions to any practical application-motivated model theory fall under one of two related, but slightly different, archetypes: (1) application-oriented engineering-standard, which is focused on the work’s practical usability (like numerical stability of an implemented algorithm or a degree of similarity of its output to a real phenomenon it models), but is less preoccupied with a formal theoretical analysis of the contribution; and (2) theoretically-oriented mathematical-standard, which is focussed on a logically impeccable theoretical analysis of one or more models, but is less preoccupied with (often even oblivious to) a practical usability and/or value of the models. In the application-motivated theories that model a given phenomenon (including the image segmentation theory, in which we are interested in), usually many engineering-standard contributions are made before a mathematical-standard work starts to appear. This seems to be caused by the initial user-induced demand for different model-based usable products and by the fact that the mathematical-standard work especially thrives at the later stage, when there is enough engineering-standard contributions to analyze. It is our belief, that the image segmentation theory has long reached the stage in its development where mathematical-standard analysis of the existing models should start to appear.

In this work, we present new tools for theoretical analysis and comparison of segmentation algorithms with the hope that they will help to initiate a greater influx of mathematical-standard contributions to image segmentation theory. To stress the value of the approach we have taken, we needed to point out different mathematical-standard weaknesses in the current image segmentation literature. This, however, does not mean that we do not appreciate the great scientific value of this literature. In particular, we have chosen to cite only the papers whose contribution we highly value and which are relevant in one way or another to this paper.

Image segmentation—the process of partitioning the image domain into meaningful object regions—is perhaps the most challenging and critical problem in image processing and analysis. Its central position in image processing comes from the fact that the delineation of objects is usually the first step in other higher level processing tasks, like image interpretation, diagnosis, analysis, visualization, virtual object manipulation, and often even registration. Image segmentation may be thought of as consisting of two related processes: recognition and delineation. Recognition is the high-level process of determining roughly the whereabouts of an object of interest in the image. Delineation is the low-level process of determining the precise spatial extent and point-by-point composition (material membership percentage) of the object in the image. The topic of this paper concerns image delineation.

General segmentation frameworks are usually broadly classified into three groups: boundary-based [14], [21], [22], [27], [31], [33], [35], [36], region-based [5], [9], [44], [48], [49], [50], [51], and hybrid [12], [25]. However, for the purpose of this paper, the more important classification comes from the mode of algorithm introduction, that is, either as discretizations of continuous models (e.g., functional optimization methods, usually implemented via level sets [31], [36], [44], including active contour [27], [29]), or as purely discrete algorithms (e.g., graph-cut [7], [8], [9], fuzzy connectedness [16], [40], [41], [51], and watershed¹ [53], [37], [45]). The first group of algorithms comes always with two theoretical constructs: the continuous model $\mathcal{M}$ and its discretization algorithm $\mathcal{A}$. On the other hand, for any algorithm $\mathcal{A}$ introduced in a purely discrete fashion, the existence and format of its continuous counterpart $\mathcal{M}$ is, in general, not clear. Our main tool for algorithm comparison will require a full description of the pairs $〈\mathcal{M}\text{,}\mathcal{A}〉$ and a good understanding of the relation between $\mathcal{M}$ and $\mathcal{A}$. Therefore, in what follows, we will argue that all reasonable segmentation algorithms, including those introduced in a purely discrete fashion, should have their continuous counterparts. We will also formalize the intuitive relation between the continuous segmentation models $\mathcal{M}$ and the related algorithms $\mathcal{A}$. It should be stressed that, even for the algorithms introduced as discretizations of continuous models, there are very few published proofs connecting in any mathematically meaningful way the models with the algorithms. (Important exceptions are the papers [6], [1], [2], [13], but see comments in the footnote #2.) Therefore, even in this class of algorithms, there is currently no mathematically sound theory of segmentation algorithms.² Thus, the most fundamental and theoretical goal of this paper is to initiate the study that will connect in a provable mathematical way the theory of continuous image segmentation models with the theory of discrete segmentation algorithms.

A practical motivation for the development of a general theoretical study of image segmentation methodologies as alluded to above is to address several gaps that currently exist in our knowledge in this subject, which are denoted (G1)–(G3) in the following: (G1) Are all different families of segmentation methods/models really fundamentally distinct or are there similarities, or even theoretical equivalences, among them? Although there are some rare attempts to compare the methods at a theoretical level (see e.g. [3], [19], [24], [34]), this is largely an uncharted territory. (G2) Segmentation research has two clearly distinct components: the practical, focused on describing efficient segmentation algorithms that can be practically implemented; and theoretical, concerning development and use of sophisticated tools of infinite (i.e., not discrete) mathematics for the purpose of describing segmentation models of idealized images. One of the peculiarities of the current state of segmentation research is that these two tracks are hardly connected in any formal way. True, the papers that start with a description of a segmentation model of idealized images usually transcribe such a model into a digital image segmentation procedure. However, all such translations are done only at an imprecise intuitive³ level, without a formal, mathematical argument. In fact, there is even no evidence of the use of any definition formally connecting idealized images (infinite objects) with their digital representations (which are finite). (G3) Another element clearly missing from current segmentation research is a set of properties that any digital segmentation algorithm must or should have. For example, it seems desirable to have the output of any reasonable segmentation algorithm to be reasonably stable if it is fed with the digital approximations of the same idealized image with better and better resolution. It would be also desirable for the segmentation output to remain reasonably unchanged when applied to the same resolution digital representations of the same idealized image that was rotated and/or shifted. (This latter aspect becomes important when we keep in mind that, in many areas such as medical imaging, there is no guarantee that the same object with subtle and fine features will be digitized in the same manner in repeated scans/digitizations, although some empirical evaluations of segmentation algorithms have assessed the variability in repeated scans.) So far, there is little research done systematically addressing points (G1)–(G3), especially for the algorithms that were not motivated by idealized image segmentation models. This paper is a first attempt to fill some of these gaps via the general theory proposed in Section 2.

In the fuzzy connectedness, FC, framework [51], a fuzzy topological construct, called fuzzy connectedness, characterizes how the spatial elements (abbreviated as spels) of an image hang together to form an object. This construct is arrived at roughly as follows. A function called affinity is defined on the image domain; the strength of affinity between any two spels depends on how close the spels are spatially and how similar their intensity-based properties are in the image. Affinity is intended to be a local relation. A global fuzzy relation called fuzzy connectedness is induced on the image domain by affinity as follows. For any two spels c and d in the image domain, all possible paths connecting c and d are considered. Each path is assigned a strength of fuzzy connectedness which is simply the minimum of the affinities of consecutive spels along the path. The level of fuzzy connectedness between c and d is considered to be the maximum of the strengths of all paths between c and d. For segmentation purposes, FC is utilized in several ways as described below. See [49] for a review of the different FC definitions and how they are employed in segmentation and applications.

In absolute FC (abbreviated AFC) [51], the support of a segmented object is considered to be the maximal set of spels, containing one or more seed spels, within which the level of FC is at or above a specific threshold. To obviate the need for a threshold, relative FC (or RFC) [40] was developed by letting all objects in the image to compete simultaneously via FC to claim membership of spels in their sets. To avoid treating the core aspects of an object (that are very strongly connected to its seeds) and the peripheral subtle aspects (that may be less strongly connected to the seeds) in the same footing, an iterative refinement strategy is devised in iterative RFC (or IRFC) [16], [50]. RFC and IRFC can be viewed as graph cut optimizations for appropriate cost functions [19]. The FC family of methods developed to date consists of various combinations of absolute, relative, and iterative FC. In this paper we will study (in Section 4.1) only the AFC algorithm, considered with a gradient based affinity. Note that gradient based affinity is a generalized affinity notion, in a format introduced and examined in [17], [18]. The other forms of FC algorithms will be examined within the general framework of Section 2 in our future work.

The level set method refers to the specific model of an evolving front (surface or curve) in a time dependent manner and to the numerical algorithm tracking such propagating fronts. The model and the associated narrow band propagation algorithm were introduced in 1985 by Sethian [43] which made their way into image segmentation research in 1995 with paper [31]. The popularity of the level set method in segmentation tasks led to a multitude of research papers, as exemplified by the books [38], [39], [44]. Although the level set method in image segmentation is nowadays more often used indirectly to solve the PDE optimizing the segmentation cost functions (see e.g., [15], [32], [52]), the original segmentation algorithms are still studied [46], [47]. Therefore, in this paper, for the purpose of using the theoretical framework for comparing methods, we have chosen the level set method with front propagation, because of its popularity, and FC, because of our familiarity with it. Our choice of the particular algorithms with each of these classes is not for their popularity or power (factually, the level set algorithms are far more popular than the FC algorithms), but rather to make our point that the algorithms introduced via very different mathematical tools can be asymptotically equivalent.

Our general theoretical framework and its variations are described in Sections 2 A general image segmentation framework, 3 Some application-driven variations of the general definitions. The application of the theory to the analysis of a particular model of FC [51] and to a comparison of its algorithms with the level set delineation algorithm of [31] is presented in Section 4. (An attempt at expressing this level set algorithm [31] without PDE can also be found in [46], [47].) Although in this paper we focus only on the algorithms of [51], [31] for a theoretical comparison, the general framework can be utilized to compare any methods in the literature. In Section 5, we present some practical segmentation examples to illustrate the equivalence proved in Section 4 and state our conclusions.