The three-class ideal observer for univariate normal data: Decision variable and ROC surface properties

Abstract

Although a fully general extension of ROC analysis to classification tasks with more than two classes has yet to be developed, the potential benefits to be gained from a practical performance evaluation methodology for classification tasks with three classes have motivated a number of research groups to propose methods based on constrained or simplified observer or data models. Here we consider an ideal observer in a task with underlying data drawn from three univariate normal distributions. We investigate the behavior of the resulting ideal observer’s decision variables and ROC surface. In particular, we show that the pair of ideal observer decision variables is constrained to a parametric curve in two-dimensional likelihood ratio space, and that the decision boundary line segments used by the ideal observer can intersect this curve in at most six places. From this, we further show that the resulting ROC surface has at most four degrees of freedom at any point, and not the five that would be required, in general, for a surface in a six-dimensional space to be non-degenerate. In light of the difficulties we have previously pointed out in generalizing the well-known area under the ROC curve performance metric to tasks with three or more classes, the problem of developing a suitable and fully general performance metric for classification tasks with three or more classes remains unsolved.

Introduction

We are working to extend the well-known observer performance evaluation methodology of receiver operating characteristic (ROC) analysis (Egan, 1975, Metz, 1978) to classification tasks with three or more classes. This could conceivably be of benefit, for example, in a medical decision-making task in which a region of a patient image must be characterized as containing a malignant lesion, a benign lesion, or only normal tissue (Chan et al., 2003, Edwards et al., 2004); or in a task in which multiple types of abnormality or defect must be distinguished from each other and from benign or normal cases (He, Metz, Tsui, Links, & Frey, 2006).

Unfortunately, a fully general extension of ROC analysis to classification tasks with more than two classes has yet to be developed, for the following reasons. It is known that the performance of an observer in a classification task with $N$ classes ($N\ge 2$) can be completely described by a set of $N^2-N$ conditional error probabilities (Van Trees, 1968), and that the performance of the ideal observer (that which minimizes Bayes risk (Van Trees, 1968)) is completely characterized by an ROC hypersurface in which these conditional error probabilities depend on a set of $N^2-N-1$ decision criteria (Edwards, Metz, & Kupinski, 2004). Although analytic expressions for the ideal observer’s conditional error probabilities given reasonable models for the underlying observational data have been worked out in the two-class case (Metz & Pan, 1999), this has not yet been accomplished in a fully general manner for tasks with three or more classes. Furthermore, we have shown that an obvious generalization of the area under the ROC curve (AUC) does not in fact yield a useful performance metric in tasks with three or more classes (Edwards, Metz, & Nishikawa, 2005). More recently, we showed that complicated constraining relationships exist among the decision boundaries themselves for the ideal observer (Edwards & Metz, 2005). These constraining relationships appear to imply that it is highly unlikely that analytical expressions for the conditional error probabilities in terms of the decision criteria can be developed which are as simple to interpret as those for the two-class task.

Despite the difficulties just described, the potential benefits to be gained from a practical performance evaluation methodology for classification tasks with three classes have motivated a number of research groups to propose such methods. These practical methods reduce the number of degrees of freedom used to describe the observer’s performance, either by implicitly leaving the remaining degrees of freedom out of the analysis, or by explicitly imposing restrictions on the form of the observer’s decision rule or on the set of decision criteria used by the observer. An example of the former would be the work of Mossman (1999) and of Dreiseitl, Ohno-Machado, and Binder (2000), which proposed reporting performance only in terms of the three “sensitivities” (conditional classification rates for correctly identifying the three classes) in a three-class task, while examples of the latter would include the works of Scurfield, 1996, Scurfield, 1998, of Chan et al. (2003) and Sahiner, Chan, and Hadjiiski (2006), of He et al. (2006) and He and Frey (2006), and of Schubert, Thorsen, and Oxley (2011). (Recently, He et al. have pointed out (He, Gallas, & Frey, 2010) that, given complete analytic knowledge of a three-class ideal observer’s “sensitivity” ROC surface, whose coordinates are the three correct classification probabilities, one can recover the probability density functions (pdfs) of the pair of likelihood ratios used by that observer. These pdfs, in turn, can be used to recover a complete description of the observer’s performance, including the misclassification ROC surface. However, the practical consequences of this theoretical result have yet to be fully investigated; and for the present, we are concerned here with a situation in which the underlying data pdfs are already assumed known, obviating the need for such an intermediate step via measurement of the sensitivity surface.)

The first work by Scurfield mentioned above (Scurfield, 1996), and that of Schubert et al. (2011), are of particular interest because they consider models which assume not only a particular form for the observer’s decision rule, but also that the underlying observational data, for which decisions are being made, are univariate. Such a univariate model is worth exploring for both theoretical and pragmatic reasons. Since a univariate model will be more tractable, in general, than a multivariate model, the former can serve as an important step in the development of a fully general classification and performance evaluation methodology; although the development of a suitable performance metric and evaluation methodology is beyond the scope of the present work, this, which is our long-term goal. More pragmatically, a variety of medical decision-making tasks can be envisioned in which the underlying data consist of a single variable: the result of a particular blood test; or the size of a lesion in a context where other features of the lesion can be ignored, such as a skin test for allergies or tuberculosis. The psychophysical task considered by Scurfield was the ability of an observer to distinguish monotonic auditory stimuli into “low tone”, “medium tone”, and “high tone” classes based on perceived frequency; the clinical task considered by Schubert et al. was to distinguish chronic allograft nephropathology from normal kidney function and proteinuria based on protein markers present in patient urine.

The decision rule of interest here, considered in both papers just mentioned is defined by a pair of thresholds on the underlying data; observations less than the lower threshold are assigned to one class, those between the two thresholds to a second class, and those above the higher threshold to a third class. This is a straightforward generalization of two-class classification models (such as the conventional (Metz, Herman, & Shen, 1998) and proper (Metz & Pan, 1999) binormal models; recall that the latter is motivated by an ideal observer approach to the underlying data model of the former) in which a single threshold is used to separate univariate data into two classes. While more complicated observer behavior is certainly conceivable, it is perhaps surprising that the three-class ideal observer itself is not limited to such a decision strategy. For normally distributed underlying data, in particular, we show here that, depending on the values of its five decision criteria and the distributional parameters of the underlying data, the three-class ideal observer operates by dividing the underlying data axis into as many as seven regions (i. e., by dividing the underlying data axis with as many as six thresholds). This is by no means intended as a criticism of the works already cited, which do not directly address the ideal observer; the particular observer models developed in those works were explicitly assumed to apply to arbitrary decision strategies, as opposed to specifically ideal observer decision strategies.

A univariate decision variable model in a task with more than two classes was investigated also by Kijewski, Swensson, and Judy (1989). However, evaluation was performed only using standard two-class ROC analysis on pairs of classes.

In Section 2, we review the theory of the three-class ideal observer, and develop explicit expressions for the ideal observer’s likelihood ratio decision variables in the case of normally distributed underlying data. In particular, because the two decision variables are functions of univariate data, it is shown that one of those decision variables can be expressed as a relation of the other (a curve, open or closed, in the decision variable space). Because the form of this relation between the decision variables is determined by the relative means and variances of the three pdfs of the underlying data, conditional on membership in each of the three classes, we consider three distinct situations: in Section 3, the case of all three variances being equal; in Section 4, the case in which two of the three variances are equal, and the third is distinct from them; and in Section 5, the case in which all three variances are distinct. In Section 6, these results are used to develop closed-form expressions for the ROC operating points of the ideal observer under consideration. In Section 7, we summarize some general properties of parametric surfaces that will be of use in subsequent sections. In Section 8, we apply these properties to the ROC surface at hand to obtain constraints on the number of degrees of freedom of that surface. In Section 9, we discuss the most salient qualitative features of the preceding results; and finally, the conclusions of the present work are given in Section 10.