Instances of computational optimal recovery: Refined approximability models

Abstract

Models based on approximation capabilities have recently been studied in the context of Optimal Recovery. These models, however, are not compatible with overparametrization, since model- and data-consistent functions could then be unbounded. This drawback motivates the introduction of refined approximability models featuring an added boundedness condition. Thus, two new models are proposed in this article: one where the boundedness applies to the target functions (first type) and one where the boundedness applies to the approximants (second type). For both types of models, optimal maps for the recovery of linear functionals are first described on an abstract level before their efficient constructions are addressed. By exploiting techniques from semidefinite programming, these constructions are explicitly carried out on a common example involving polynomial subspaces of $\mathcal{C}[-1,1]$.

Introduction

The objective of this article is to uncover practical methods for the optimal recovery of functions available through observational data when the underlying models based on approximability allow for overparametrization. To clarify this objective and its various challenges, we start with some background on traditional Optimal Recovery. Typically, an unknown function $f$ defined on a domain $D$ is observed through point evaluations $y_1=f\left(x_1\right),\dots ,y_m=f\left(x_m\right)$ at distinct points $x_1,\dots ,x_m\in D$. More generally, an unknown object $f$, simply considered as an element of a normed space $X$, is observed through

$$y_i={\ell }_i\left(f\right),i\in [1:m],$$

where ${\ell }_1,\dots ,{\ell }_m$ are linear functionals defined on $X$. We assume here that these data are perfectly accurate — we refer to the companion article [5] for the incorporation of observation error. The data is summarized as $y=L\left(f\right)$, where the linear map $L:X\to {\mathbb{R}}^m$ is called observation operator. Based on the knowledge of $y\in {\mathbb{R}}^m$, the task is then to recover a quantity of interest $Q\left(f\right)$, where throughout this article $Q:X\to \mathbb{R}$ is assumed to be a linear functional. The recovery procedure can be viewed as a map $R$ from ${\mathbb{R}}^m$ to $\mathbb{R}$, with no concern for its practicability at this point.

Besides the observational data (which is also called a posteriori information), there is some a priori information coming from an educated belief about the properties of realistic $f$’s. It translates into the assumption that $f$ belongs to a model set $\mathcal{K}\subseteq X$. The choice of this model set is of course critical. When the $f$’s indeed represent functions, it is traditionally taken as the unit ball with respect to some norm that characterizes smoothness. More recently, motivated by parametric partial differential equations, a model based on approximation capabilities has been proposed in [2]. Namely, given a linear subspace $V$ of $X$ and a threshold $\epsilon >0$, it is defined as

$$\mathcal{K}={\mathcal{K}}_{V,\epsilon }≔\{f\in X:{\mathrm{dist}}_X(f,V)\le \epsilon \}.$$

This model set is also implicit in many numerical procedures and in machine learning.

Whatever the selected model set, the performance of the recovery procedure $R:{\mathbb{R}}^m\to \mathbb{R}$ is measured in a worst-case setting via the (global) error of $R$ over $\mathcal{K}$, i.e., 

$$e_{\mathcal{K},Q}(L,R)≔\underset{f\in \mathcal{K}}{sup}|Q\left(f\right)-R\left(L\left(f\right)\right)|.$$

Obviously, one is interested in optimal recovery maps $R^{\mathrm{opt}}:{\mathbb{R}}^m\to \mathbb{R}$ minimizing this worst-case error, i.e., such that

$$e_{\mathcal{K},Q}(L,R^{\mathrm{opt}})=\underset{R:{\mathbb{R}}^m\to \mathbb{R}}{inf}{e}_{\mathcal{K},Q}(L,R).$$

This infimum is called the intrinsic error of the observation map $L$ (for $Q$ over $\mathcal{K}$). It is known, at least since Smolyak’s doctoral dissertation [13], that there is a linear functional among the optimal recovery maps as soon as the set $\mathcal{K}$ is symmetric and convex, see e.g. [10, Theorem 4.7] for a proof. The practicality of such a linear optimal recovery map is not automatic, though. For the approximability set (2), Theorem 3.1 of [4] revealed that such a linear optimal recovery map takes the form $R^{\mathrm{opt}}:y\in {\mathbb{R}}^m\mapsto {\sum }_{i=1}^m{a}_i^{\mathrm{opt}}{y}_i$, where $a^{\mathrm{opt}}\in {\mathbb{R}}^m$ is a solution to

$$\underset{a\in {\mathbb{R}}^m}{\mathrm{minimize}}\Vert Q-\sum _{i=1}^m{a}_i{\ell {}_i\Vert }_{X^{\ast }}\text{subject to }\sum _{i=1}^m{a}_i{\ell }_i\left(v\right)=Q\left(v\right)\text{for all }v\in V,$$

an optimization problem that can be solved for $X=\mathcal{C}\left(D\right)$ in exact form when the observation functionals are point evaluations (see [4]) and in approximate form when they are arbitrary linear functionals (see [5] or Section 3.2).

The approximability set (2), however, presents some important restrictions. Suppose indeed that there is some nonzero $v\in ker\left(L\right)\cap V$. Then, for a given $f_0\in \mathcal{K}$ observed through $y=L\left(f_0\right)\in {\mathbb{R}}^m$, any $f_t≔f_0+tv$, $t\in \mathbb{R}$, is both model-consistent (i.e., $f_t\in \mathcal{K}$) and data-consistent (i.e., $L\left(f_t\right)=y$), so that the local error at $y$ of any recovery map $R:{\mathbb{R}}^m\to \mathbb{R}$ satisfies

$$e_{\mathcal{K},Q}^{\mathrm{loc}}(L,R\left(y\right))≔\underset{\genfrac{}{}{0ex}{}{f\in \mathcal{K}}{L\left(f\right)=y}}{sup}|Q\left(f\right)-R\left(y\right)|\ge \underset{t\in \mathbb{R}}{sup}|Q\left(f_t\right)-R\left(y\right)|=\underset{t\in \mathbb{R}}{sup}|(Q\left(f_0\right)-R\left(y\right))+tQ\left(v\right)|,$$

which is infinite whenever $Q\left(v\right)\ne 0$. Thus, for the optimal recovery problem to make sense under the approximability model (2), independently of the quantity of interest $Q$, one must assume that $ker\left(L\right)\cap V=\left\{0\right\}$. By a dimension argument, this imposes

$$n≔dim\left(V\right)\le m.$$

In other words, we must place ourselves in an underparametrized regime for which the number $n$ of parameters describing the model does not exceed the number $m$ of data. This contrasts with many current studies, especially in the field of Deep Learning, which emphasize the advantages of overparametrization. In order to incorporate overparametrization in the optimal recovery problem under consideration, we must then restrict the magnitude of model- and data-consistent elements. A glaring strategy consists in altering the approximability set (2). We do so in two different ways, namely by considering a bounded approximability set of the first type, i.e., 

$$\mathcal{K}={\mathcal{K}}_{V,\epsilon ,\kappa }^{\mathrm{I}}≔\left\{f\in X:{\mathrm{dist}}_X(f,V)\le \epsilon \text{ and }{\Vert f\Vert }_X\le \kappa \right\},$$

and a bounded approximability set of the second type, i.e., 

$$\mathcal{K}={\mathcal{K}}_{V,\epsilon ,\kappa }^{\mathrm{II}}≔\left\{f\in X:\exists v\in V\text{ with }{\Vert f-v\Vert }_X\le \epsilon \text{ and }{\Vert v\Vert }_X\le \kappa \right\}.$$

We will start by analyzing the second type of bounded approximability sets in Section 2 by formally describing the optimal recovery maps before revealing on a familiar example how the associated minimization problem is tackled in practice. The main ingredient in essence belongs to the sum-of-squares techniques from semidefinite programming. Next, we will analyze the first type of bounded approximability sets in Section 3. We will even formally describe optimal recovery maps over more general model sets consisting of intersections of approximability sets. On the prior example, we will again reveal how the associated minimization problem is tackled in practice. This time, the main ingredient in essence belongs to the moment techniques from semidefinite programming. In view of this article’s emphasis on computability issues, all of the theoretical constructions are illustrated in a reproducible matlab file downloadable from the author’s webpage.

Before delving into technicalities, we stress that this article considers a scenario where $L$ is fixed, meaning that the user is not free to select favorable observation functionals ${\ell }_1,\dots ,{\ell }_m$. As such, we are not concerned with the minimal number $m$ of observations needed to achieve a prescribed accuracy. In other words, we do not investigate the complexity of the problem. But replacing ${\mathcal{K}}_{V,\epsilon }$ by ${\mathcal{K}}_{V,\epsilon ,\kappa }^{\mathrm{I}}$ or ${\mathcal{K}}_{V,\epsilon ,\kappa }^{\mathrm{II}}$ is actually akin to a strategy which is popular in complexity studies: substituting a refined model set for a model set that was initially too rich. This strategy can transform an intractable multivariate problem into a tractable one, see [12] for a typical example. For our refined approximability sets, complexity inquiries would be interesting to pursue, especially in order to uncover more qualitative statements on the benefits (or lack thereof) of overparametrization. This would require, for a start, to precisely estimate the worst-case error (4) minimized over $L$, and in particular its dependence on the number of variables of multivariate functions $f\in X$. However, this is beyond the scope of this article, whose focus is placed on the computability of the optimal recovery maps.