Sharpness Estimation of Combinatorial Generalization Ability Bounds for Threshold Decision Rules

Abstract

This article is devoted to the problem of calculating an exact upper bound for the functionals of the generalization ability of a family of one-dimensional threshold decision rules. An algorithm is investigated that solves the stated problem and is polynomial in the total number of samples used for training and validation and in the number of training samples. A theorem is proved for calculating an estimate for the functional of expected overfitting and an estimate for the error rate of the method for minimizing empirical risk on a validation set. The exact bounds calculated using the theorem are compared with the previously known quick-to-compute upper bounds so as to estimate the orders of overestimation of the bounds and to identify the bounds that could be used in real problems.

APPENDIX

Proof of Theorem 5. Let us write the expected overfitting formula and permute summation signs in it,

$$ EOF = {\binom L\ell }^{-1}\thinspace \sum _{X \in \left [\mathbb {X}\right ]^l}\thinspace \sum _{p=0}^{P} \thinspace \left [\mu X = a_p\right ] \: \delta \left (a_p, X\right ) = {\binom L\ell }^{-1} \thinspace \sum _{p = 0}^{P}\thinspace \sum _{X \in \left [\mathbb {X}\right ]^l}\thinspace \left [\mu X = a_p\right ] \: \delta (a_p, X). $$

Consider the set of partitions $(X, \bar {X})$ with fixed values of $t$ and $e$,

$$ t = |X \cap \mathbb {D}|, \quad e = n(a_p, X \cap \mathbb {D}).$$

The set of admissible values of $(t, e)$ is $\Psi _p$ according to (8).

Set $s = n(a_p, X \cap \mathbb {N})$. The constraints $s + t\leqslant l$ and $s \leqslant m$ imply the upper bound for the parameter $s$ in (7).

Since the number of classifier errors $a_p$ on the validation sample is

$$ n\left (a_p, \bar {X}\right ) = n\left (a_p, \mathbb {X}\right ) - n\left (a_p, X\right )=n\left (a_p, \mathbb {X}\right ) - n\left (a_p, X \cap \mathbb {D}\right ) - n\left (a_p, X \cap \mathbb {N}\right ),$$

the overfitting of the classifier $a_p$ for given $s$ and $e$ can be represented in the form

$$ \delta \left (a_p, X\right ) = \frac {1}{L - \ell }\thinspace n\left (a_p, \bar {X}\right ) - \frac {1}{\ell }\thinspace n\left (a_p, X\right ) = \frac {1}{L - \ell }\thinspace \left (n\left (a_p, \mathbb {X}\right ) - \left (s + e\right )\right ) - \frac {1}{\ell } \thinspace \left (s + e\right ).$$

It was proved in [24] that whether the condition $\mu X =a_p$ is satisfied is independent of the choice of partitions of the set $\mathbb {N}$, and it was shown that the number of partitions of the set $\mathbb {N}$ for given $t$ and $s$ is equal to $\binom ms \binom {L - P - m}{\ell - t - s}$; this implies the statement of the theorem. The proof of Theorem 5 is complete. $\quad \blacksquare$

Proof of Theorem 6. Using the Hoeffding inequality [30], with probability $\varepsilon$ one can estimate the deviation $\eta = \eta (\varepsilon )$ of overfitting from its expectation,

$$ \delta \left (\mu X, \bar {X}\right ) \leq EOF\left (\mu \right ) + \eta \left (\varepsilon \right ), $$

where the deviation is $\eta = \sqrt {-2 \ln \tfrac {\varepsilon }{2}}$.

Then the error rate of the classifier selected by the PERM algorithm on the validation sample can be estimated directly via the error rate on the learning sample and the overfitting expectation,

$$ \nu \left (\mu X, \bar {X}\right ) = \nu \left (\mu X, \bar {X}\right ) + \delta \left (\mu X, \bar {X}\right ) \leq \nu \left (\mu X, X\right ) + EOF\left (\mu \right ) + \eta \left (\varepsilon \right );$$

this justifies inequality (9). The latter and Lemmas 1 and 2 imply inequality (10). The proof of Theorem 6 is complete. $\quad \blacksquare$