The Isowarp: The Template-Based Visual Geometry of Isometric Surfaces

Abstract

Registration maps or warps form a key element in Shape-from-Template (SfT). They relate the template with the input image, which contains the projection of the deformed surface. Recently, it was shown that isometric SfT can be solved analytically if the warp and its first-order derivatives are known. In practice, the warp is recovered by interpolating a set of discrete template-to-image point correspondences. This process relies on smoothness priors but ignores the 3D geometry. This may produce errors in the warp and poor reconstructions. In contrast, we propose to create a 3D consistent warp, which technically is a very challenging task, as the 3D shape variables must be eliminated from the isometric SfT equations to find differential constraints for the warp only. Integrating these constraints in warp estimation yields the isowarp, a warp 3D consistent with isometric SfT. Experimental results show that incorporating the isowarp in the SfT pipeline allows the analytic solution to outperform non-convex 3D shape refinement methods and the recent DNN-based SfT methods. The isowarp can be properly initialized with convex methods and its hyperparameters can be automatically obtained with cross-validation. The isowarp is resistant to 3D ambiguities and less computationally expensive than existing 3D shape refinement methods. The isowarp is thus a theoretical and practical breakthrough in SfT.

Appendices

Derivation of the Image Embedding

We assume that the image plane is at $z=1$ in camera coordinates, which is achieved by working in retinal coordinates. The perspective projection of a point (x, y, z) is then given by:

$$\begin{aligned} \Pi _{p}:\qquad {\mathbb {R}}^{3}\setminus \left\{ (x,y,z)\in {\mathbb {R}}^3\;|\;z=0\right\}&\rightarrow {\mathbb {R}}^2\nonumber \\ \left( x,y,z\right) ^\top&\mapsto \left( \frac{x}{z},\frac{y}{z}\right) ^\top . \end{aligned}$$

(25)

The inverse of the restriction $\Pi _{p}|_{{\mathcal {S}}}:{\mathcal {S}} \rightarrow {\mathbb {R}}^2$ of $\Pi _{p}$ is the image embedding. It forms a depth based parametrization of the surface $({\mathcal {I}},X_{i})$ expressed in terms of the depth function $\rho :{\mathbb {R}}^2 \rightarrow {\mathbb {R}}$:

$$\begin{aligned} X_{i}(u',v') = \rho (u',v') \left( u',v',1\right) ^\top , \end{aligned}$$

(26)

where $u'$ and $v'$ represent the image coordinates. Alternatively to $\rho$, we define the Euclidean distance between the camera’s projection origin and the surface point as ${\tilde{\rho }}:{\mathbb {R}}^2 \rightarrow {\mathbb {R}}$:

$$\begin{aligned} {\tilde{\rho }}(u',v')=\rho (u',v')\zeta (u',v'), \end{aligned}$$

(27)

where $\zeta (u',v')=\sqrt{1+u'^2+v'^2}$. The perspective parametrization $({\mathcal {I}},X_i)$ can be then expressed in terms of ${\tilde{\rho }}$ as:

$$\begin{aligned} X_{i}(u',v') = \frac{{\tilde{\rho }}(u',v')}{\zeta (u',v')}\left( u',v',1\right) ^\top . \end{aligned}$$

(28)

Now, we can define the surface ${\mathcal {S}}$ from the template parametrization domain ${\mathcal {U}}$ by composing the previous parametrization $X_i$ and the warp function $\eta$ as follows:

$$\begin{aligned} {\bar{X}}_i(u,v) = X_i \circ \eta = \frac{{\tilde{\rho }}(\eta (u,v))}{\zeta (\eta (u,v))}\left( \eta (u,v),1\right) ^\top , \end{aligned}$$

(29)

where u and v are template domain coordinates.

Defining the depth function ${\bar{\rho }}:{\mathbb {R}}^2 \rightarrow {\mathbb {R}}$ as the depth function ${\tilde{\rho }}$ in u, v coordinates by the composition ${\bar{\rho }} = {\tilde{\rho }}\circ \eta$, we obtain:

$$\begin{aligned} {\bar{X}}_i(u,v) = X_i \circ \eta = \frac{{\bar{\rho }}(u,v)}{\zeta (\eta (u,v))}\left( \eta (u,v),1\right) ^\top . \end{aligned}$$

(30)

Working with the parametrization $({\mathcal {U}},{\bar{X}}_i)$ of ${\mathcal {S}}$ has two principal advantages. First, it allows us to compute the first fundamental form, also known as the metric tensor, over the same parametrization domain as the template, which is essential to obtain the isowarp equations. Second, it greatly simplifies these equations.

Derivation of the Isowarp Equations

We give Matlab code to establish the Isowarp equations (11). These equations are too lengthy to be reproduced fully expanded. However we recall that, importantly, they depend on the known template and the unknown warp $\eta$ only. More specifically, they are quadratic of the second-order in $\eta$.