1 Introduction

Real symmetric \(n \times n\) matrices are identified with quadratic forms on \({\mathbb {R}}^n\), and they form a vector space \({\mathbb {S}}^n\) of dimension \(\left( {\begin{array}{c}n+1\\ 2\end{array}}\right) \). We write \({\mathbb {S}}^n_+\) for the subset of quadratic forms that are nonnegative on \({\mathbb {R}}^n\). This is a full-dimensional closed semialgebraic convex cone in \({\mathbb {S}}^n\), known as the PSD cone. Its elements are identified with positive semidefinite matrices. The PSD cone is self-dual with respect to the trace inner product \(A \circ B := {\mathrm {trace}}(A B)\) for \(A, B \in {\mathbb {S}}^n\).

Given any linear subspace \({\mathcal {L}} \) of \({\mathbb {S}}^n\), we consider the linear projection \(\pi _{\mathcal {L}}: {\mathbb {S}}^n \rightarrow {\mathcal {L}}^\vee \) that is dual to the inclusion \({\mathcal {L}} \subset {\mathbb {S}}^n\). Here, \({\mathcal {L}}^\vee = \mathrm{Hom}({\mathcal {L}},{\mathbb {R}})\) denotes the vector space dual of \({\mathcal {L}}\). We are interested in the image \( \pi _{\mathcal {L}}({\mathbb {S}}^n_+)\) of the PSD cone under this map. These objects can be written in coordinates as follows. If \(\{A_1,A_2,\ldots ,A_k\} \subset {\mathbb {S}}^n\) is a basis of \({\mathcal {L}}\) then our map is

$$\begin{aligned} \pi _{\mathcal {L}} \,:\, {\mathbb {S}}^n \,\rightarrow \, {\mathbb {R}}^k \,, \,\, X \,\mapsto \,\left( A_1 \circ X,\, A_2 \circ X , \ldots , \,A_k \circ X \right) . \end{aligned}$$
(1)

While the PSD cone \({\mathbb {S}}^n_+\) is closed in \({\mathbb {S}}^n\), its image under the map \( \pi _{\mathcal {L}}\) may not be.

Example 1

Fix \(n=k=2\). Let \({\mathcal {L}} \) be the linear space spanned by the two quadratic forms \(q_1 = x_1^2\) and \(q_2 = 2x_1 x_2\). The corresponding symmetric \(2 \times 2\) matrices are

$$A_1 \,= \,\left(\begin{array}{ll} 1 & 0 \\ 0 & 0 \end{array}\right) \quad {\mathrm{and}} \quad A_2 \, = \, \left(\begin{array}{ll} 0 & 1 \\ 1 & 0 \end{array}\right) $$

The linear map \(\pi _{\mathcal {L}}\) projects the 3-dimensional space \({\mathbb {S}}^2\) into the plane \({\mathbb {R}}^2\) via

$$\begin{aligned} X = \left(\begin{array}{ll} x_{11} & x_{12} \\ x_{12} & x_{22} \end{array}\right) \, \,\mapsto \, \, ( A_1 \circ X,\, A_2 \circ X ) \, = \, ( \,x_{11}, \,2 x_{12} \,). \end{aligned}$$

The image of the PSD cone \({\mathbb {S}}^2_+\) under this map into \({\mathbb {R}}^2\) is not closed. We find that

$$\begin{aligned} \ \pi _{\mathcal {L}} ( {\mathbb {S}}^2_+) \,\,=\, \, \left\{ \,(z_1,z_2) \in {\mathbb {R}}^2 \,: z_1 > 0 \,\,\,{\mathrm{or}}\,\,\, z_1 = z_2 = 0\, \right\} . \end{aligned}$$

The failure of the image to be closed reflects the fact that strong duality can fail in semidefinite programming (SDP). A thorough study of this phenomenon was undertaken by Pataki in [12, 15,16,17]. Basics on SDP and its algebraic aspects can be found in [1, Chapter 1] and [13, Chapter 12]. Our subspace \({\mathcal {L}}\) plays the role of an instance of SDP, as in [13, Corollary 12.12]. Using the adjective proposed in [16], the subspace \({\mathcal {L}}\) is called bad if the cone \(\pi _{\mathcal {L}} ( {\mathbb {S}}^n_+) \) is not closed. But, just like in slang usage, “bad” can also mean “good”. Pataki derives a characterization, and he concludes that bad semidefinite programs all look the same [16].

The aim of this paper is to examine this phenomenon through the lens of algebraic geometry. The assertion of [16] that all bad instances “look the same” refers to the natural action of the group \({\mathrm{GL}}(n) \times {\mathrm{GL}}(k)\) on the domain and range of our map \(\pi _{\mathcal {L}}\) in (1). Pataki describes normal forms of bad instances \({\mathcal {L}}\) with respect to that action, to be reviewed in Sect. 2. We are here interested in the geometry of the locus of all bad instances, that is, the orbits of Pataki’s normal forms under \({\mathrm{GL}}(n) \times {\mathrm{GL}}(k)\). We pass to the Zariski closure, and study the corresponding complex projective variety. The following example is meant to illustrate how our perspective builds on and differs from that developed in [12, 15, 16] and [17].

Example 2

Fix \(n=k=2\). Let \({\mathcal {L}}\) be a general subspace in \({\mathbb {S}}^2\), with a given basis

$$\begin{aligned} q_1 \,=\, a_{11} x_1^2 + 2 a_{12} x_1 x_2 + a_{13} x_2^2 \quad {{\mathrm{and}}} \quad q_2 \,=\, b_{11} x_2^2 + 2 b_{12} x_1 x_2 + b_{13} x_2^2. \end{aligned}$$

The nature of \(\pi _{\mathcal {L}}({\mathbb {S}}_+^2)\) is determined by the resultant of these two binary quadrics:

$$\begin{aligned} R \,= \, a_{11}^2 b_{22}^2-4 a_{11} a_{12} b_{12} b_{22}-2 a_{11} a_{22} b_{11} b_{22} +4 a_{11} a_{22} b_{12}^2 + 4 a_{12}^2 b_{11} b_{22} -4 a_{12} a_{22} b_{11} b_{12}+a_{22}^2 b_{11}^2. \end{aligned}$$

Our theory in Sect. 3 implies that \(\pi _{\mathcal {L}}({\mathbb {S}}_+^2)\) is not closed in \({\mathbb {R}}^2\) if and only if \(R=0\). Furthermore, if \(R > 0\) then \(\pi ({\mathbb {S}}_+^2) \) is a closed pointed cone. Yet, if \(R < 0\) then \(\,\pi ({\mathbb {S}}_+^2) = {\mathbb {R}}^2\). If \( R > 0\) then \({\mathcal {L}}\) is spanned by two squares, \(\gamma _1 q_1 + \gamma _2 q_2 = (u_1 x_1 + u_2 x_2)^2\) and \(\delta _1 q_1 + \delta _2 q_2 = (v_1 x_1 + v_1 x_2)^2\), and we have \(\pi ({\mathbb {S}}^2_+) \,= \, \left\{ (z_1,z_2) \in {\mathbb {R}}^2 \,:\, \gamma _1 z_1 + \gamma _2 z_2 \ge 0 \,\,\,{\mathrm{and}} \,\,\, \delta _1 z_1 + \delta _2 z_2 \ge 0 \,\right\} \). If \(R=0\) then the two squares are linearly dependent and the image cone is not closed:

$$\begin{aligned} \pi ({\mathbb {S}}^2_+) \quad = \quad \left\{ \,(z_1,z_2) \in {\mathbb {R}}^2 \,:\, \gamma _1 z_1 + \gamma _2 z_2 > 0 \,\right\} \,\,\cup \,\, \left\{ (0,0) \right\} . \end{aligned}$$
(2)

In conclusion, all bad instances do look the same as Example 1. But, from an algebraic perspective, their parameter space \(\{R=0\}\) is a variety of great interest.

This article is organized as follows. Section 2 characterizes all linear subspaces \({\mathcal {L}}\) in \({\mathbb {S}}^n\) that are bad in the sense that the image cone \(\pi _{\mathcal {L}} ({\mathbb {S}}^n_+)\) is not closed. This result is due to Pataki [16, 17]. In Theorem 1 we present his linear algebra formulation in terms of block matrices. We recast this in the setting of real algebraic geometry. This is motivated by Proposition 1 which states that the bad subspaces \({\mathcal {L}}\) form a semialgebraic subset of the Grassmannian \({\mathrm{Gr}}(k,{\mathbb {S}}^n)\). In Example 7, we offer a contrast to the analogous closure question for the images of the quadratic maps \({\mathbb {R}}^n \rightarrow {\mathbb {R}}^k\) and \({\mathbb {C}}^n \rightarrow {\mathbb {C}}^k\) that also arise from our subspace \({\mathcal {L}} \simeq {\mathbb {R}}^k\) of \({\mathbb {S}}^n\).

In Sect. 3, we turn to projective geometry and study the Zariski closure \({\mathrm{Bad}}_{k,n}\) of the set of bad subspaces \({\mathcal {L}}\) inside the complex Grassmannian \({\mathrm{Gr}}(k,{\mathbb {S}}^n)\). These varieties have codimension one and they are generally reducible. Their irreducible components are the coisotropic hypersurfaces [11] of rank strata of symmetric matrices. This is the content of Theorem 2, which identifies bad projections with objects familiar from elimination theory, such as resultants, Chow forms and Hurwitz forms [7, 21]. Hyperdeterminants [7] remain in the background. Examples 8 and 9 offer detailed analyses of the cases \(n=3\) and \(n=4\).

Section 4 explains the badness of a subspace \({\mathcal {L}}\) in terms of the normal cycle of the cone \({\mathbb {S}}^n_+\) and its Zariski closure in \({\mathbb {P}}({\mathbb {S}}^n) \times {\mathbb {P}}({\mathbb {S}}^n)\). The latter is the projective normal cycle, whose irreducible components are the conormal varieties of the rank strata [1, Example 5.15]. Following [14], this encodes complementarity in SDP. Theorem 3 reveals that \({\mathcal {L}}\) is bad when the normal cycle meets \({\mathcal {L}} \times {\mathcal {L}}^\perp \). This furnishes effective algebraic tools to identify bad projections, illustrated by computations with Macaulay2 [8] in Examples 11, 12 and 13. We invite our readers to peek at Example 10 where the resultant from Example 2 is revisited.

2 How to be bad

We are interested in the subset of the real Grassmannian \({\mathrm{Gr}}(k,{\mathbb {S}}^n)\) whose points are the bad linear spaces \({\mathcal {L}}\). By definition, a space \({\mathcal {L}}\) is bad if the image cone \(\pi _{\mathcal {L}}({\mathbb {S}}^n_+)\) is not closed in \({\mathbb {R}}^k\).

Proposition 1

The set of bad linear spaces \({\mathcal {L}}\) is a semialgebraic subset of the Grassmannian \({\mathrm{Gr}}(k,{\mathbb {S}}^n)\). This set is not closed when \(n \ge 3\).

Proof

The first assertion follows from the Tarski-Seidenberg Theorem on Quantifier Elimination [1, Theorem A.49]. Indeed, the image \(\pi _{\mathcal {L}}({\mathbb {S}}^n_+)\) is a semialgebraic subset of \({\mathbb {R}}^k\), i.e., it can be described by a Boolean combination of polynomial inequalities. The dependence on \({\mathcal {L}}\) is semialgebraic, as is the statement that the image is not closed. We can eliminate the coordinates of \({\mathbb {R}}^k\) to obtain a quantifier-free formula in the Plücker coordinates of \({\mathcal {L}}\). This formula describes the desired semialgebraic subset of the real Grassmannian \({\mathrm{Gr}}(k,{\mathbb {S}}^n)\).

To see that this subset is not closed, fix \(k=2,n=3\), let t be a parameter, and consider the quadrics \(q_1 = x_1^2\) and \(q_2 = x_2^2 + t x_1 x_3\). Their span is a 2-dimensional subspace \({\mathcal {L}}_t\) in \({\mathbb {S}}^3\) for all \(t \in {\mathbb {R}}\). For \(t \not =0\), the space \({\mathcal {L}}_t\) is bad because \(\pi _{{\mathcal {L}}_t}({\mathbb {S}}^3) = \,\{z_1 > 0\} \cup \{z_1 = 0 \,\,{\mathrm{and}}\,\, z_2 \ge 0 \}\,\) is not closed. For \(t=0\), the image \(\pi _{{\mathcal {L}}_0}({\mathbb {S}}^3) = {\mathbb {R}}^2_{\ge 0}\) is closed, so \({\mathcal {L}}_0\) is good. This specifies a sequence of bad points in \({\mathrm{Gr}}(2,{\mathbb {S}}^3)\) whose limit is a good point, so our set is not closed. We note that, by Example 2, for \(n=2\) the set \(\{R = 0\}\) of bad \({\mathcal {L}}\) is closed in \({\mathrm{Gr}}(2,{\mathbb {S}}^3) \simeq {\mathbb {P}}^2\). \(\square \)

A characterization of bad subspaces was given by Pataki. Our first goal is to present his result. We consider the spectrahedral cone \(\,{\mathcal {L}} \cap {\mathbb {S}}^n_+\). This convex cone is related to our object of interest by duality:

Lemma 1

The closure of \( \,\pi _{\mathcal {L}}({\mathbb {S}}^n_+)\) is linearly isomorphic to the cone dual to \(\,{\mathcal {L}} \cap {\mathbb {S}}^n_+\).

Proof

This follows from the first statement in [19, Corollary 16.3.2], when A is the linear map \( \pi _{{\mathcal {L}}}\) and the convex set C is the PSD cone \({\mathbb {S}}^n_+\). Note that \({\mathbb {S}}^n_+\) is self-dual. \(\square \)

We now assume that \(k \ge 2\), because \({\mathcal {L}}\) is always good when \(k=1\): the one-dimensional cone \( \pi _{\mathcal {L}}({\mathbb {S}}^n_+)\) equals \( {\mathbb {R}}\) if \(\,{\mathcal {L}} \cap {\mathbb {S}}^n_+ = \{0\}\) and it is \({\mathbb {R}}_{\ge 0}\) or \({\mathbb {R}}_{\le 0}\) otherwise.

Definition 1

For a linear subspace \({\mathcal {L}}\) of \({\mathbb {S}}^n\), we fix a quadric q of maximal rank in the spectrahedral cone \({\mathcal {L}} \cap {\mathbb {S}}^n_+\). The rank of q is an invariant of \({\mathcal {L}}\), denoted \(s= s({\mathcal {L}})\) and called the spectrahedral rank of \({\mathcal {L}}\).

If \(s = n\) then \(\,{\mathcal {L}} \cap {\mathbb {S}}^n_+\) is full-dimensional and \(\pi _{\mathcal {L}}({\mathbb {S}}^n_+)\) is pointed and closed. If \(s= 0\) then \(\,{\mathcal {L}} \cap {\mathbb {S}}^n_+ = \{0\}\) and \( \pi _{\mathcal {L}}({\mathbb {S}}^n_+) = {\mathcal {L}}^\vee \), which is also closed. Thus we are mostly interested in the cases where \(0< s < n \). After a linear change of coordinates given by the action of \({\mathrm{GL}}(n)\), we may assume \(q = x_1^2 + x_2^2 + \cdots + x_s^2\). The matrix that represents a quadratic form \(v \in {\mathbb {S}}^n\) has the block structure

$$\begin{aligned} \qquad \qquad V \,=\, \begin{pmatrix} V_{11} &{} V_{12} \\ V_{12}^T &{} V_{22} \end{pmatrix}, \qquad {\mathrm{where}} \quad V_{11} \in {\mathbb {S}}^s \,\,\, {{\mathrm{and}}} \,\,\, V_{22} \in {\mathbb {S}}^{n-s}. \end{aligned}$$

The following result appears in [17, Theorems 1,2]. A subspace \({\mathcal {L}}\) is called good if it is not bad. We write \({\mathcal {L}}^\perp = \ker (\pi _{{\mathcal {L}}})\) for the orthogonal complement of \({\mathcal {L}}\) in \({\mathbb {S}}^n\).

Theorem 1

(Pataki) A linear space \({\mathcal {L}} \subseteq {\mathbb {S}}^n\) is bad if and only if there exists a quadric \(v \in {\mathcal {L}}\) whose associated matrix V satisfies \(V_{22} \in {\mathbb {S}}^{n-s}_+\) and \(\,\mathrm {im}\,(V_{12}^T) \not \subseteq \mathrm {im}\,(V_{22})\). The space \({\mathcal {L}}\) is good if and only if there exists a positive definite matrix \(\,U \in {\mathbb {S}}^{n-s}_+\) such that \(\begin{pmatrix} 0 &{} 0 \\ 0 &{} U \end{pmatrix}\in {\mathcal {L}}^\perp \) and, for all matrices \(V \in {\mathcal {L}}\), the condition \(V_{22} = 0\) implies \(V_{12} = 0\).

We now present an alternative version of this result. Since q is positive semidefinite, it is a sum of squares \(l_1^2 + \ldots + l_s^2\) of linear forms \(l_1, \dots , l_s\), and \(I_L\) is the ideal generated by \(l_1, \ldots , l_s\). For instance, if \(n \ge 3\) and \(q = (x_1-x_2)^2 + (x_1 + x_2)^2 \) then \(I_{\mathcal {L}} = \langle x_1,x_2 \rangle \). We consider the inclusions of linear spaces

$$\begin{aligned} {\mathcal {L}} \cap I_{\mathcal {L}}^2 \, \,\, \subseteq \,\,\, {\mathcal {L}} \cap I_{\mathcal {L}} \,\,\, \subseteq \,\,\, {\mathcal {L}} \,\,\subseteq \,\, {\mathbb {S}}^n \,=\, {\mathbb {R}}[x_1,\ldots ,x_n]_2 . \end{aligned}$$

The first space \({\mathcal {L}} \cap I_{\mathcal {L}}^2\) is the linear span of the spectrahedon \(\,{\mathcal {L}} \cap {\mathbb {S}}^n_+\), while the second space \({\mathcal {L}} \cap I_{\mathcal {L}}\) also records tangent directions relative to the PSD cone. To illustrate the inclusions, we consider the bad plane in Example 1, where \(q = x_1^2\), \(I_{\mathcal {L}} = \langle x_1 \rangle \) and \(v = x_1 x_2 \in I_{\mathcal {L}} \backslash I_{\mathcal {L}}^2\). We already know that the existence of such a pair (q, v) characterizes non-closed projections.

Corollary 1

A linear space of quadrics \({\mathcal {L}} \subset {\mathbb {S}}^n\) is bad if and only if \(s({\mathcal {L}}) \ge 1 \) and

$$\begin{aligned} \quad s({\mathcal {L}}) + s({\mathcal {L}}^\perp )\, < \, n \quad {\mathrm{or}} \quad {\mathcal {L}} \cap I_{\mathcal {L}}^2 \, \subsetneq \,{\mathcal {L}} \cap I_{\mathcal {L}}. \end{aligned}$$
(3)

Proof

We set \(s = s({\mathcal {L}})\) and assume \(I_{{\mathcal {L}}} = \langle x_1, \dots , x_s \rangle \). If \(s=0\) then \(\,{\mathcal {L}} \cap {\mathbb {S}}^n_+\,= \,\{0\}\) and thus, by Lemma 1, the closure of \( \pi _{\mathcal {L}}({\mathbb {S}}^n_+)\) is equal to \( \{0\}^\perp = {\mathcal {L}}^\vee \simeq {\mathbb {R}}^k\). But, since \(\pi _{\mathcal {L}}({\mathbb {S}}^n_+)\) is convex, this implies that \(\pi _{\mathcal {L}}({\mathbb {S}}^n_+) \) equals \( {\mathcal {L}}^\vee \), so \({\mathcal {L}}\) is good. Thus, we can now assume \(s \ge 1\).

We claim that the two conditions in the disjunction in (3) are the negations of the two conditions in the conjunction that characterizes goodness in the last statement of Theorem 1. Indeed, since two matrices in \({\mathbb {S}}^n_+\) have trace inner product equal to zero if and only if their matrix product is the zero matrix, we always have \(s({\mathcal {L}}^\perp ) \le n-s\). The equality \(s({\mathcal {L}}^\perp ) = n-s\) holds if and only if there is a positive definite \((n-s) \times (n-s)\) matrix U as in Theorem 1. Next, consider any matrix V that would correspond to a quadratic form in \( ({\mathcal {L}} \cap I_{{\mathcal {L}}}) \backslash ( {\mathcal {L}} \cap I_{{\mathcal {L}}}^2)\). Containment in \( {\mathcal {L}} \cap I_{{\mathcal {L}}}\) means that \(V_{22} = 0\), and non-containment in \({\mathcal {L}} \cap I_{{\mathcal {L}}}^2\) means that \(V_{12} \not = 0\). We conclude that Corollary 1 is the contrapositive of the last statement of Theorem 1. \(\square \)

Example 3

Let \(n=k=3\) and let \({\mathcal {L}}\) be spanned by the span of the quadratic forms

$$\begin{aligned} \begin{matrix} q_1 &{} = &{} -52 x_1^2\,+\,412 x_1 x_2+472 x_1 x_3+462 x_2^2+1164 x_2 x_3+750 x_3^2\,, \\ q_2 &{} = &{} -101 x_1^2+435 x_1 x_2+480 x_1 x_3+518 x_2^2+1307 x_2 x_3+853 x_3^2\, , \\ q_3 &{} = &{} -55 x_1^2\,+\,362 x_1 x_2+482 x_1 x_3+434 x_2^2+ 1166 x_2 x_3+772 x_3^2\, . \end{matrix} \end{aligned}$$

These dense quadrics are chosen to hide the properties of \({\mathcal {L}}\). They are identified with symmetric \(3 \times 3\)-matrices \(A_1,A_2,A_3\), so that \(q_i = (x_1,x_2,x_3) A_i (x_1,x_2,x_3)^T\) for \(i=1,2,3\). To reveal the nature of \({\mathcal {L}}\), we display the elements

$$\begin{aligned} \begin{matrix} 7 q_1 - 4 q_2 - 2 q_3 &{} = &{} 6 \cdot (5 x_1 + 7 x_2 + 7 x_3)^2, \qquad \qquad \qquad \qquad \\ 17 q_1 - 14 q_2 + 2 q_3 \,\,&{} = &{} \quad 42 \cdot (5 x_1 + 7 x_2 + 7 x_3) (2 x_1 + 5 x_2 + 8 x_3) .\,\, \end{matrix} \end{aligned}$$

Hence \(I_{\mathcal {L}} = \langle 5 x_1 + 7 x_2 + 7 x_3 \rangle \). The space \(\,{\mathcal {L}} \cap I_{\mathcal {L}}^2 \,\) is one-dimensional and spanned by the square above, but \( \,{\mathcal {L}} \cap I_{\mathcal {L}} \,\) is two-dimensional. Our two linear combinations form a basis. The theorem shows that the cone \(\pi _{\mathcal {L}}({\mathbb {S}}^3_+)\) is not closed. It is the union of an open half-space in 3-space, together with a line through the origin in the plane that bounds the half-space.

Example 4

Let \(n=k=3\) and let \({\mathcal {L}}\) be the span of the quadratic forms

$$\begin{aligned} q_1 = x_1^2, q_2 = x_2^2 + 2x_1x_3, q_3 = 2x_2x_3 \end{aligned}$$

Hence \(I_{\mathcal {L}} = \langle x_1 \rangle \). The space \({\mathcal {L}} \cap I_{\mathcal {L}}^2 = {\mathcal {L}} \cap I_{\mathcal {L}}\) is spanned by \(q_1\). The orthogonal complement \({\mathcal {L}}^\perp \) is spanned by

$$\begin{aligned} q_1' = 2x_1x_2, q_2' = 2x_1x_3 - 2x_2^2, q_3' = x_3^2 \end{aligned}$$

Hence \(I_{\mathcal {L}} = \langle x_3 \rangle \), so \(s({\mathcal {L}}) = s({\mathcal {L}}^\perp ) = 1\) and their sum is \(2 < n=3\). The cone \(\pi _{\mathcal {L}}({\mathbb {S}}^3_+)\) is the union of an open half-space and a half-line that bounds it.

We now turn the section title around and we focus on how to be good. For our best case scenario, we assume \(s({\mathcal {L}}) =n\). This means that \({\mathcal {L}}\) intersects the interior of the PSD cone \({\mathbb {S}}^n_+\). Hence the intersection \( {\mathcal {L}} \cap {\mathbb {S}}^n_+\) is a full-dimensional pointed cone in \({\mathcal {L}} \simeq {\mathbb {R}}^k\). Its convex dual \(({\mathcal {L}} \cap {\mathbb {S}}^n)^\vee \) is a full-dimensional pointed cone in the dual space \({\mathcal {L}}^\vee \simeq {\mathbb {R}}^k\). Lemma 1 and Theorem 1 imply that this closed dual cone is precisely our projection of the PSD cone:

$$\begin{aligned} \pi _{\mathcal {L}} ({\mathbb {S}}^n_+) \,\,\, = \,\,\, ({\mathcal {L}} \cap {\mathbb {S}}^n_+)^\vee . \end{aligned}$$
(4)

Suppose that \({\mathcal {L}}\) is generic among points in \({\mathrm{Gr}}(k,{\mathbb {S}}^n)\) that satisfy \(s({\mathcal {L}}) = n\). The image cone (4) is a generic spectrahedral shadow, in the sense of [20]. The boundary of the image is an algebraic hypersurface that can have multiple irreducible components, one for each matrix rank r in the Pataki range; see e.g. [5, Lemma 5] and [20, Theorem 1.1]. The degree of the rank r component is a positive integer, denoted \(\delta (k,n,r)\), that is known as the algebraic degree of semidefinite programming. These degrees play a major role in our main result, which is Theorem 2. We refer to [20, Table 1] for explicit numbers, and to the bibliographies of [2, 5, 14, 20] and [22] for additional references. For instance, for projections of the PSD cone \({\mathbb {S}}^3_+\) into dimensions \(k=3\) and \(k=4\), we find that \(\delta (2,3,2) = 6\) and \(\delta (3,3,r) = 4\) for \(r=1,2\). The following two subspaces exhibit the generic good behavior.

Example 5

(\(n=k=3\)) Let \({\mathcal {L}}\,\) be the space spanned by the rank two quadrics

$$\begin{aligned} q_1 \,=\, x_1^2 \,+\, (x_2+x_3)^2 \, ,\quad q_2 \,=\, x_2^2 \,+\, (x_1+x_3)^2 \, , \quad q_3 \,=\, x_3^2 \,+\, (x_1+x_2)^2 . \end{aligned}$$

Their sum is positive definite, so \(s({\mathcal {L}}) = 3\). This specific linear space \({\mathcal {L}}\) appeared in [22] as an illustration for linear concentration models in statistics. In that application, the three-dimensional cone (4) serves as the cone of sufficient statistics of the model. Its boundary is an irreducible surface of degree six, defined by the polynomial \(H_{\mathcal {L}}\) shown in [22, Example 1.1]. That surface is the cone over the plane sextic curve shown in red on the left in Fig. 1. For a discussion of this curve in the context of semidefinite programming see [13, Example 12.5].

Fig. 1
figure 1

The cones over the convex bodies shown on the left \((k=3)\) and right \((k=4)\) are the images of the 6-dimensional cone \({\mathbb {S}}^3_+\) under projections \(\pi _{\mathcal {L}}\) defined by good subspaces \({\mathcal {L}}\)

Full size image

Example 6

(\(n=3,k=4\)) Let \({\mathcal {L}}\,\) be the space spanned by the rank one quadrics

$$\begin{aligned} q_1 = x_1^2 \, ,\quad q_2 = x_2^2 \, ,\quad q_3= x_3^2 \, ,\quad q_4 = (x_1+x_2+x_3)^2 . \end{aligned}$$

Their sum is positive definite, so \(s({\mathcal {L}}) = 3\). The 4-dimensional cone \({\mathcal {L}} \cap {\mathbb {S}}^3_+\) is the cone over a 3-dimensional convex body known as elliptope and shown in [13, Fig. 1.1]. The cone (4) is bounded by four hyperplanes and a quartic threefold in \({\mathbb {R}}^4\). These correspond respectively to the four circles and the Roman surface that bounds the green body on the right in Fig. 1.

In the literature, there has been a discrepancy between studies in convex algebraic geometry, like [2, 20], and how semidefinite programming is actually used. The former has focused on generic figures, while the latter often concerns special instances from combinatorial optimization. For such scenarios, strong duality can fail, thus motivating works like [10, 12] and [17]. The present paper aims to reconcile these perspectives. We study special scenarios through the lens of algebraic geometry, by highlighting cases that are generic among the bad ones.

We close this section with a brief exploration of another connection to algebraic geometry. Namely, we consider the restriction of \(\pi _{{\mathcal {L}}}\) to the set of rank one matrices in \({\mathbb {S}}^n_+\). This set comprises the extreme rays of the cone \({\mathbb {S}}^n_+\), and it coincides with the image of the map \([{\mathcal {L}}]_{\mathbb {R}}:{\mathbb {R}}^n \rightarrow {\mathbb {S}}^n_+\) that takes a vector \(\mathbf{y}\) to the rank one matrix \(\mathbf{y}^T \cdot \mathbf{y}\). It is therefore equivalent to study the quadratic map \({\mathbb {R}}^n \rightarrow {\mathbb {R}}^k\) defined by evaluating the quadrics \(q_1,\ldots ,q_k\) that span \({\mathcal {L}}\). The image \({\mathrm{im}}([{\mathcal {L}}]_{\mathbb {R}})\) of this map in \({\mathbb {R}}^k\) coincides with the image of rank one matrices under \(\pi _{{\mathcal {L}}}\). What is this image, and under what conditions on \({\mathcal {L}}\) is it closed? To answer these questions for a small instance, one can apply the method of Cylindrical Algebraic Decomposition [4]. In our experiments we used the implementation Resolve in Mathematica.

For an algebraic geometer, it is natural to first pass to the algebraic closure of the field \({\mathbb {R}}\) and to consider the map \([{\mathcal {L}}_{\mathbb {C}}]:{\mathbb {C}}^n \rightarrow {\mathbb {C}}^k\) over the complex numbers. Our questions remain as above. What is the image \({\mathrm{im}}([{\mathcal {L}}]_{\mathbb {C}})\) of this map in \({\mathbb {C}}^k\), and under which conditions on \({\mathcal {L}}\) is it closed? We know from [13, Theorem 4.23] that \({\mathrm{im}}([{\mathcal {L}}]_{\mathbb {C}})\) is closed if \({\mathcal {L}}\) has no zeros in \({\mathbb {P}}^{n-1}\). To answer these questions for any given instance, we can apply the algorithm due to Harris, Michałek and Sertöz [9]. Our experiments used their implementation in Macaulay2 [8].

We now compare the closure property for the three sets \(\pi _{\mathcal {L}}({\mathbb {S}}^n_+)\), \({\mathrm{im}}([{\mathcal {L}}]_{\mathbb {R}})\) and \({\mathrm{im}}([{\mathcal {L}}]_{\mathbb {C}})\). In each example we display a basis for \({\mathcal {L}}\). One easily finds cases when all three sets are closed, like \((x_1^2,x_2^2)\), or where none is closed, like \((x_1^2,x_1x_2)\). The following maps are more interesting.

Example 7

(Disagreements) Fix \(n=3\). Four instances are listed by dimension k:

  1. (2)

    \({\mathcal {L}} = (x_1^2+x_2^2,x_1x_3)\). Here \(\pi _{\mathcal {L}}({\mathbb {S}}^3_+) = {\mathrm{im}}([{\mathcal {L}}]_{\mathbb {R}}) = \{z_1 > 0\} \cup \{z_1=z_2=0\}\) is not closed, but the complexification \([{\mathcal {L}}_{\mathbb {C}}]:{\mathbb {C}}^3 \rightarrow {\mathbb {C}}^2\) is onto. In particular, \( {\mathrm{im}}([{\mathcal {L}}]_{\mathbb {C}}) = {\mathbb {C}}^2\) is closed.

  2. (3)

    \({\mathcal {L}} = (x_1x_2,x_1x_3,x_2x_3)\). Here \(\pi _{\mathcal {L}}({\mathbb {S}}^3_+) = {\mathbb {R}}^3\) is closed because the off-diagonal entries of a positive semidefinite matrix can be arbitrary. However, \( {\mathrm{im}}([{\mathcal {L}}]_{\mathbb {R}}) \) and \( {\mathrm{im}}([{\mathcal {L}}]_{\mathbb {C}}) \) are not closed. Their images lack all points that have precisely one coordinate zero.

  3. (4)

    \({\mathcal {L}} = (x_1^2+x_2^2+x_3^2,x_1 x_2, x_2 x_3)\). Here, \(\pi _{\mathcal {L}}({\mathbb {S}}^3_+) = {\mathrm{im}}([{\mathcal {L}}]_{\mathbb {R}}) \) is a closed pointed convex cone in \({\mathbb {R}}^3\). The Macaulay2 code in [9] outputs \(\,{\mathrm{im}}([{\mathcal {L}}]_{\mathbb {C}}) \,=\, {\mathbb {C}}^3 \setminus \left\{ (0,\lambda ,\lambda \cdot i)\,:\, \lambda \in {\mathbb {C}}\backslash \{0\} \right\} \). This is not closed. However, since \(i = \sqrt{-1}\), none of the missing points is real.

  4. (5)

    \({\mathcal {L}} = (x_1^2-x_2^2,x_3^2,x_1x_2,x_1x_3,x_2x_3)\). Here \(\pi _{\mathcal {L}}({\mathbb {S}}^3_+) \) is not closed because \({\mathcal {L}}^\perp = (x_1^2+x_2^2)\) is a semidefinite ray. The complex image \( {\mathrm{im}}([{\mathcal {L}}]_{\mathbb {C}}) \) is the affine threefold in \({\mathbb {C}}^5\) defined by \( z_2 z_3=z_4 z_5\) and \( z_1 z_2+z_5^2 = z_4^2\), and \( {\mathrm{im}}([{\mathcal {L}}]_{\mathbb {R}}) \) is its set of real points. Both are closed.

3 Coisotropic hypersurfaces

In this section we study the projective variety \({\mathrm{Bad}}_{k,n}\). By definition, this is the Zariski closure in the complex Grassmannian \({\mathrm{Gr}}(k,{\mathbb {S}}^n)\) of the set of bad subspaces \({\mathcal {L}}\). Our goal is to characterize \({\mathrm{Bad}}_{k,n}\) in terms of objects that are familiar to algebraic geometers. We focus on subvarieties of Grassmannians called higher associated varieties that are introduced in Sect. 3.2.E of the book on Discriminants, Resultants and Multidimensional Determinants by Gel’fand, Kapranov and Zelevinsky [7]. They were further studied in recent work of Kohn [11] who calls them coisotropic varieties. We shall adopt that name and the notation in [11].

We write \({\mathbb {P}}({\mathbb {S}}^n) \simeq {\mathbb {P}}^{\left( {\begin{array}{c}n+1\\ 2\end{array}}\right) -1}\) for the space of symmetric \(n \times n\) matrices. Every point \({\mathcal {L}}\) in \({\mathrm{Gr}}(k,{\mathbb {S}}^n)\) represents a \((k-1)\)-dimensional linear subspace \({\mathbb {P}}{\mathcal {L}}\) of \({\mathbb {P}}({\mathbb {S}}^n)\). Let \({\mathcal {Y}}\) be a variety of codimension c in \({\mathbb {P}}({\mathbb {S}}^n)\). For a regular point \(X \in {\mathrm{Reg}}({\mathcal {Y}})\), we write \(T_X {\mathcal {Y}}\,\) for the tangent space of \({\mathcal {Y}}\) at X. The i-th coisotropic variety of \({\mathcal {Y}}\) is

$$\begin{aligned} {\mathrm {Ch}}_i({\mathcal {Y}}) \,\,:= \,\,\overline{\left\{ \, {\mathcal {L}} \in {\mathrm {Gr}}(c+i, {{\mathbb {S}}}^n) \,\mid \,\exists \,X \in {\mathrm {Reg}}({\mathcal {Y}}) \cap {\mathbb {P}}{\mathcal {L}}\,:\, \dim ({\mathbb {P}}{\mathcal {L}} \cap T_X {\mathcal {Y}}) \ge i \, \right\} }. \end{aligned}$$
(5)

The bar denotes Zariski closure in the Grassmannian. Note that \(T_X {\mathcal {Y}}\) has codimension c while \({\mathbb {P}}{\mathcal {L}} \) has dimension \(c+i-1\). The expected dimension of their intersection is \(i-1\). Hence \({\mathrm{Ch}}_i({\mathcal {Y}})\) is a proper subvariety of \({\mathrm {Gr}}(c+i, {\mathbb {S}}^n)\), and we expect this to be a hypersurface. Coisotropic varieties are generalizations of Chow forms. We shall see this in our examples below. The abbreviation “Ch” can thus stand both for Chow from and for coisotropic hypersurface.

The following theorem accomplishes the goal stated above. We write \({\mathcal {X}}_s\) for the subvariety of \({\mathbb {P}}({\mathbb {S}}^n)\) given by all symmetric \(n \times n\)-matrices of rank \(\le s\). We set \(c = \left( {\begin{array}{c}n-s+1\\ 2\end{array}}\right) = \mathrm {codim}({\mathcal {X}}_s)\).

Theorem 2

The bad subvariety \(\,{\mathrm{Bad}}_{k,n}\) has codimension one in \(\,{\mathrm{Gr}}(k,{\mathbb {S}}^n)\). It is the union of the irreducible coisotropic hypersurfaces \(\,{\mathrm{Ch}}_{k-c}({\mathcal {X}}_s)\), where s runs over integers such that

$$\begin{aligned} \left( {\begin{array}{c}n-s+1\\ 2\end{array}}\right) \, \,<\,\, k \,\,\le \,\, \left( {\begin{array}{c}n+1\\ 2\end{array}}\right) -\left( {\begin{array}{c}s+1\\ 2\end{array}}\right) . \end{aligned}$$
(6)

The degree of the irreducible polynomial in Plücker coordinates that defines \({\mathrm{Ch}}_{k-c}({\mathcal {X}}_s)\) is the algebraic degree of semidefinite programming, which is denoted by \(\,\delta (k,n,s)\).

The proof will be presented later in this section. Our first task is to make this statement understandable by defining all ingredients, and then prove it near the end of this section. We recall (e.g. from [13, Chapter 5]) that the Grassmannian \({\mathrm{Gr}}(k,{\mathbb {S}}^n)\) is embedded, via the Plücker embedding, in a projective space of dimension \(\left( {\begin{array}{c}\left( {\begin{array}{c}n+1\\ 2\end{array}}\right) \\ k\end{array}}\right) -1\). The coordinates we use for \({\mathrm{Gr}}(k,{\mathbb {S}}^n)\) are described in [11, Sect. 2.1]. The primal Stiefel coordinates are the entries of matrices \(A_1,\ldots ,A_k\) that span \({\mathcal {L}}\). The dual Stiefel coordinates are matrix entries for a basis of \({\mathcal {L}}^\perp \). If we vectorize these basis elements and write them as the rows of a matrix with \(\left( {\begin{array}{c}n+1\\ 2\end{array}}\right) \) columns, then the maximal minors of this matrix are the Plücker coordinates of \({\mathcal {L}}\).

Fix any irreducible variety \({\mathcal {Y}}\) in \({\mathbb {P}}({\mathbb {S}}^n) \simeq {\mathbb {P}}^{\left( {\begin{array}{c}n+1\\ 2\end{array}}\right) -1}\). Its projectively dual variety \({\mathcal {Y}}^\vee \) parametrizes hyperplanes that are tangent to \({\mathcal {Y}}\) at some regular point. Note that \({\mathcal {Y}}\) and \({\mathcal {Y}}^\vee \) live in the same ambient space \({\mathbb {P}}({\mathbb {S}}^n)\), since \({\mathbb {S}}^n\) is identified with its dual via the trace inner product. We set \(c = {\mathrm{codim}}({\mathcal {Y}})\) and \(d = {\mathrm{dim}}({\mathcal {Y}}^\vee )\). Following [11] and (5) above, we write \({\mathrm{Ch}}_i({\mathcal {Y}})\) for the i-th coisotropic variety of \({\mathcal {Y}}\). This is an irreducible subvariety of the Grassmannian \({\mathrm{Gr}}(k,{\mathbb {S}}^n)\), where \(k=c+i\). The points of \({\mathrm{Ch}}_i({\mathcal {Y}})\) are linear subspaces \({\mathcal {L}}\) that have non-transversal intersection with the tangent space at some point of \({\mathcal {Y}}\). Kohn [11] follows the seminal work of Gel’fand, Kapranov and Zelevinsky [7] in developing a theory of coisotropic varieties. She proves in [11, Corollary 6] that \({\mathrm{Ch}}_i({\mathcal {Y}})\) is a hypersurface if and only if \(c \le k \le d + 1\). In that case, the degree of its equation in Plücker coordinates equals

$$\begin{aligned} \qquad {\mathrm{degree}}( {\mathrm{Ch}}_i({\mathcal {Y}})) \,\, = \,\, \delta _i({\mathcal {Y}}) \,:=\, \hbox {the }i\hbox {-th polar degree of }{\mathcal {Y}}. \end{aligned}$$
(7)

This appears in [11, Theorem 9]. The duality formula in [11, Theorem 20] states

$$\begin{aligned} {\mathrm{Ch}}_i({\mathcal {Y}}) \,\simeq \, {\mathrm{Ch}}_{d-c+1-i}({\mathcal {Y}}^\vee ). \end{aligned}$$
(8)

This isomorphism is equality if we identify \({\mathrm{Gr}}(k,{\mathbb {S}}^n)\) with \({\mathrm{Gr}}\left( \left( {\begin{array}{c}n+1\\ 2\end{array}}\right) -k,{\mathbb {S}}^n\right) \) given by \({\mathcal {L}} \mapsto {\mathcal {L}}^\perp \). In addition, the polar degree \(\delta _i({\mathcal {X}})\) is nonzero if and only if \(i \le d - c + 1\).

We now apply these considerations to the determinantal variety \({\mathcal {Y}}={\mathcal {X}}_s\), the codimension of which equals \(c = \left( {\begin{array}{c}n-s+1\\ 2\end{array}}\right) \). It is known that \(({\mathcal {X}}_s)^\vee = {\mathcal {X}}_{n-s}\) by [7, Proposition I.1.4.11] or [14, Proposition 12]. Hence \(d = {\mathrm{dim}}({\mathcal {X}}_s^\vee ) = {\mathrm{dim}}({\mathcal {X}}_{n-s}) = \left( {\begin{array}{c}n+1\\ 2\end{array}}\right) -1 - \left( {\begin{array}{c}s+1\\ 2\end{array}}\right) \). Hence, by [11, Corollary 6], \({\mathrm{Ch}}_k({\mathcal {X}}_s)\) is a hypersurface if and only if \(\left( {\begin{array}{c}n-s+1\\ 2\end{array}}\right) \, \,\le \,\, k \,\,\le \,\, \left( {\begin{array}{c}n+1\\ 2\end{array}}\right) -\left( {\begin{array}{c}s+1\\ 2\end{array}}\right) \). This is almost the same as (6), which is known as the Pataki range [5, 14, 20]. However, the minimal value for k in the Pataki range is disallowed in Theorem 2. The coisotropic hypersurface for that minimal value of k is the Chow form \({\mathrm{Ch}}_0 ({\mathcal {X}}_s) \). This does not contribute to \({\mathrm{Bad}}_{k, n}\).

Before proving Theorem 2, let us explore its implications for matrices of small size. For \(n=2\), the only interesting case is \(k=2\). This was studied in Example  2, where the resultant R was written in primal Stiefel coordinates on \({\mathrm{Gr}}(2,{\mathbb {S}}^2) = {\mathbb {P}}^2\). The dual Stiefel coordinates are the usual coordinates \((y_0:y_1:y_2)\) on \({\mathbb {P}}^2\), which here agree with the Plücker coordinates:

$$\begin{aligned} y_0 = 2 (a_{12} b_{22} - a_{22} b_{12}) \, , \quad y_1 = a_{22} b_{11} -a_{11} b_{22} \, ,\quad y_2 = 2 (a_{11} b_{12} - a_{12} b_{11}). \end{aligned}$$

The Veronese curve in \({\mathbb {P}}^2\) with equation \(R = y_0 y_2 - y_1^2 \) equals \(\, {\mathrm{Ch}}_0({\mathcal {X}}_1^\vee ) \simeq {\mathrm{Ch}}_1({\mathcal {X}}_1)\).

Example 8

(\(n=3\)) We discuss the bad varieties for \(k=2,3,4,5\). Theorem 2 states that \({\mathrm{Bad}}_{k,3}\) is irreducible and equal to the hypersurface \({\mathrm{Ch}}_{k-c}({\mathcal {X}}_s)\) in \({\mathrm{Gr}}(k,{\mathbb {S}}^3) = {\mathrm{Gr}}(k,6)\). The inequalities (6) imply that \(s=2\) and \(c=1\) for \(k=1,2\), and \(s=1\) and \(c=3\) for \(k=3,4\).

The hypersurface \({\mathrm{Bad}}_{2,3}\) has degree 6 in the 8-dimensional Grassmannian \({\mathrm{Gr}}(2,{\mathbb {S}}^3) \subset {\mathbb {P}}^{14}\). Its equation is the classical tact invariant of two ternary quadrics \(q_1\) and \(q_2\). The tact invariant vanishes if and only if the conics \(\{q_1=0\}\) and \(\{q_2=0\}\) are tangent in \({\mathbb {P}}^2\). When written in the \(12=6+6\) entries of the matrices \(A_1\) and \(A_2\), the tact invariant is a sum of 3210 terms of total degree 12. This is the Hurwitz form of the Veronese surface in \({\mathbb {P}}({\mathbb {S}}^3) = {\mathbb {P}}^5\). The formula in Plücker coordinates has degree six, and it appears explicitly in [21, Example 2.7].

The hypersurface \({\mathrm{Bad}}_{3,3} = {\mathrm{Ch}}_2({\mathcal {X}}_2) \simeq {\mathrm{Ch}}_0({\mathcal {X}}_1)\) has degree 4 in \({\mathrm{Gr}}(3,{\mathbb {S}}^3) \subset {\mathbb {P}}^{19}\). It is the Chow form of the Veronese embedding of \({\mathbb {P}}^2\) into \({\mathbb {P}}^5= {\mathbb {P}}({\mathbb {S}}^3)\). This is the resultant of three ternary quadrics \(q_1,q_2,q_3\) that span \({\mathcal {L}}\). When written in terms of their \(18=6+6+6\) coefficients, this resultant has 21894 terms. We invite our readers to check that this resultant vanishes at the specific instance \((q_1,q_2,q_3)\) in Example 3. Indeed, those three quadrics have two common zeros in \({\mathbb {P}}^2\).

The sextic hypersurface \({\mathrm{Bad}}_{4,3} = {\mathrm{Ch}}_1 ({\mathcal {X}}_1)\) agrees with \({\mathrm{Bad}}_{2,3} = {\mathrm{Ch}}_1({\mathcal {X}}_2)\) under the identification of \({\mathrm{Gr}}(4,{\mathbb {S}}^3)\) with \({\mathrm{Gr}}(2,{\mathbb {S}}^3)\). If we replace the 15 Plücker coordinates \(p_{ij}\) in [21, Example 2.7] with the complementary maximal minors of a \(4 \times 6\)-matrix, then we get an equation of degree 24 in the entries of a basis \(A_1,A_2,A_3,A_4\) of \({\mathcal {L}}\). The hypersurface \({\mathrm{Bad}}_{5,3} = {\mathrm{Ch}}_2({\mathcal {X}}_1)\) has degree three in \({\mathrm{Gr}}(5,{\mathbb {S}}^3) \simeq {\mathbb {P}}^{5}\). It is the determinant hypersurface \({\mathcal {X}}_2 = {\mathcal {X}}_1^\vee \) itself. Indeed, a 5-dimensional subspace \({\mathcal {L}}\) is bad if and only if the line \({\mathcal {L}}^\perp \) is spanned by a positive semidefinite matrix of rank \(\le 2\).

Example 9

(\(n=4\)) The bad hypersurfaces are irreducible for \(n \le 3\). The smallest reducible cases arise for \(n=4\), with \(k=4\) and \(k=7\). Namely, we find

$$\begin{aligned} {\mathrm{Bad}}_{4,4} \,= \,{\mathrm{Ch}}_3({\mathcal {X}}_3) \,\cup \, {\mathrm{Ch}}_1({\mathcal {X}}_2) \quad {\mathrm{and}} \quad {\mathrm{Bad}}_{7,4} \,= \, {\mathrm{Ch}}_4({\mathcal {X}}_2) \,\cup \,{\mathrm{Ch}}_1({\mathcal {X}}_1) . \end{aligned}$$

We discuss these four coisotropic hypersurfaces and their dual interpretations.

  • \({\mathrm{Ch}}_3({\mathcal {X}}_3) \simeq {\mathrm{Ch}}_0({\mathcal {X}}_1)\) has degree 8 in \({\mathrm{Gr}}(4,{\mathbb {S}}^4)\). The subspace \({\mathcal {L}} = {\mathbb {R}}\{q_1,q_2,q_3,q_4\}\) has a common zero \({\mathbf {y}}\) in \({\mathbb {P}}^3\), i.e. \({\mathcal {L}}^\perp \) contains a rank one matrix, namely \(\mathbf{y}^T\cdot \mathbf{y}\). For a generic bad instance \({\mathcal {L}}\) in this family, a basis of \({\mathcal {L}}^\perp \) is given by

    $$\begin{aligned} q_1&= x_1^2 + x_2^2 + x_3^2, \\ q_2&= \ell _1 x_1 + \ell _2 x_2 + \ell _3 x_3, \\ q_3&= \ell _4 x_1 + \ell _5 x_2 + \ell _6 x_3, \\ q_4&= \ell _7 x_1 + \ell _8 x_2 + \ell _9 x_3. \end{aligned}$$

    In this example, \({\mathbf {y}} = (0:0:0:1)\) and \(\ell _1,\ell _2,\ldots ,\ell _9\) are generic linear forms.

  • \({\mathrm{Ch}}_1({\mathcal {X}}_2) \simeq {\mathrm{Ch}}_3({\mathcal {X}}_2)\) has degree 30 in \({\mathrm{Gr}}(4,{\mathbb {S}}^4)\). In this case, both the zero-dimensional scheme \({\mathbb {P}}{\mathcal {L}} \cap {\mathcal {X}}_2\) and the surface \({\mathbb {P}}{\mathcal {L}}^\perp \cap {\mathcal {X}}_2\) are singular. Writing \(\ell _1, \dots , \ell _6\) for generic linear forms, a bad instance in this family is

    $$\begin{aligned} q_1 = x_1^2 + x_2^2 \, ,\,\,\, q_2 = \ell _1 x_1 + \ell _2 x_2 \, , \,\,\, q_3 = \ell _3 x_1 + \ell _4 x_2 + x_3^2 - x_4^2 \, , \,\,\, q_4 = \ell _5 x_1 + \ell _6 x_2 + x_3 x_4 . \end{aligned}$$

    The zero-dimensional scheme \({\mathbb {P}}{\mathcal {L}} \cap {\mathcal {X}}_2\) has length 10 in \(\,{\mathbb {P}}({\mathbb {S}}^4) \simeq {\mathbb {P}}^9\). The quadric \(q_1\) is a point of multiplicity two in that scheme.

  • \({\mathrm{Ch}}_4({\mathcal {X}}_2) \simeq {\mathrm{Ch}}_0({\mathcal {X}}_2)\) has degree 10 in \({\mathrm{Gr}}(7,{\mathbb {S}}^4)\). Here, the threefold \({\mathbb {P}}{\mathcal {L}} \cap {\mathcal {X}}_2\) is singular and \({\mathcal {L}}^\perp \) contains a rank two matrix. That matrix is a singular point of the plane quartic curve \({\mathbb {P}}{\mathcal {L}}^\perp \cap {\mathcal {X}}_3\). For a generic bad instance \({\mathcal {L}}\) of this kind, a basis of \({\mathcal {L}}^\perp \) is given by

    $$\begin{aligned} q_1 = x_3^2 + x_4^2 \, ,\,\,\, q_2 = x_1^2 - x_2^2 + \ell _1 x_1 + \ell _2 x_2 \, ,\,\,\, q_3 = x_1x_2 + \ell _3 x_1 + \ell _4 x_2. \end{aligned}$$

    Here \(\ell _1, \ell _2, \ell _3, \ell _4\) are binary linear forms in the two unknowns \(x_3\) and \( x_4\).

  • \({\mathrm{Ch}}_1({\mathcal {X}}_1) \simeq {\mathrm{Ch}}_2({\mathcal {X}}_3)\) has degree 16 in \({\mathrm{Gr}}(7,{\mathbb {S}}^4)\). Both the zero-dimensional scheme \({\mathbb {P}}{\mathcal {L}} \cap {\mathcal {X}}_1\) and the plane quartic \({\mathbb {P}}{\mathcal {L}}^\perp \cap {\mathcal {X}}_3\) are singular. The singular point is a rank three matrix. For a generic bad instance \({\mathcal {L}}\) of this kind, a basis of \({\mathcal {L}}^\perp \) is given by

    $$\begin{aligned} q_1 = x_2^2 + x_3^2 + x_4^2 \, ,\,\,\, q_2 = \ell _1 x_1 + q_2' \, ,\,\,\, q_3 = \ell _2 x_1 + q_3', \end{aligned}$$

    where \(\ell _1, \ell _2\) are linear forms in \(x_2, x_3, x_4\), and \(q_2', q_3'\) are quadrics in \(x_2, x_3, x_4\).

We note that the two irreducible components of \({\mathrm{Bad}}_{7,4}\) are precisely the irreducible factors \(\mathbf{P}\) and \(\mathbf{M}\) of the Vinnikov discriminant of a ternary quartic. This was discussed in [18, Theorem 7.5] and in [6, Remark 22]. The three matrices A, B, C that give the determinantal representation of quartics in [18] span our space \({\mathcal {L}}^\perp \). Fløystad, Kileel and Ottaviani [6] present an explicit formula for \({\mathrm{Ch}}_4({\mathcal {X}}_2)\) as the Pfaffian of a skewsymmetric \(20 \times 20\)-matrix.

Proof of Theorem 2

We already saw above that \({\mathrm{Ch}}_{k-c}({\mathcal {X}}_s)\) is a hypersurface whenever (6) holds. The degree of this coisotropic hypersurface is the i-th polar degree of \({\mathcal {X}}_s\) by (7), where \(i=k-c\). That degree equals \(\delta (k,n,s)\) by [14, Theorem 13].

In what follows we prove the assertions about the bad variety \({\mathrm{Bad}}_{k,n}\) that are stated in the first two sentences of Theorem 2. Let \(I_{{\mathcal {L}}}\) and \(I_{{\mathcal {L}}^\perp }\) be defined as in the paragraph prior to Corollary 1. The integers k, n are fixed throughout. We begin by assuming that s is in the range (6). We first show that every bad subspace \({\mathcal {L}} \in {\mathrm{Gr}}(k, {\mathbb {S}}^n)\) with \(0 < s({\mathcal {L}}^\perp ) \le n-s\) and \(0 < s({\mathcal {L}}) \le s\) is in the coisotropic hypersurface \({\mathrm{Ch}}_{k-c}({\mathcal {X}}_s)\). We shall proceed in two steps.

Step 1: Consider a linear space \({\mathcal {L}} \subset {\mathbb {S}}^n\) with \(s({\mathcal {L}}) = s\) and \(s({\mathcal {L}}^\perp ) = n-s\). Then \({\mathcal {L}}\) is bad if and only if \({\mathcal {L}} \cap I_{\mathcal {L}}^2 \subsetneq {\mathcal {L}} \cap I_{\mathcal {L}}\), by Corollary 1. Points \({\mathcal {L}}\) such that \({\mathcal {L}} \cap I_{\mathcal {L}}^2 = {\mathcal {L}} \cap I_{\mathcal {L}}\) lie in a proper subvariety of the Grassmannian \({\mathrm {Gr}}(k, {\mathbb {S}}^n)\). This means that the set of bad instances intersects the following set (9) in a Zariski dense subset:

$$\begin{aligned} \left\{ \,{\mathcal {L}} \in {\mathrm {Gr}}(k, {\mathbb {S}}^n)\, \mid \, s({\mathcal {L}}) = s \text { and } s({\mathcal {L}}^\perp ) = n-s \right\} . \end{aligned}$$
(9)

We may assume, after a change of coordinates, that \({\mathcal {L}}\) contains the diagonal matrix

$$\begin{aligned} X \,\, = \,\, {\mathrm {diag}}(\underbrace{1, \dots , 1}_{s}, 0, \dots , 0)\,\, \in \,\,{\mathrm {Reg}}({\mathcal {X}}_s), \end{aligned}$$

and that \({\mathcal {L}}^\perp \) contains the special diagonal matrix

$$\begin{aligned} Y \,\, = \,\, {\mathrm {diag}}(0, \dots , 0, \underbrace{1, \dots , 1}_{n-s}). \end{aligned}$$

Let \(H = T_X({\mathcal {X}}_s)\) denote the tangent space of the determinantal variety \({\mathcal {X}}_s\) at X. This is the linear space of codimension \(c = \left( {\begin{array}{c}n-s+1\\ 2\end{array}}\right) \) in \({\mathbb {P}}({\mathbb {S}}^n)\) defined by \(x_{ij} = 0\) for \(i >s\) and \(j > s\). The matrix Y satisfies \(Y \in H^\perp \). Hence, for a generic subspace \({\mathcal {L}}\) with \(X \in {\mathcal {L}}\) and \(Y \in {\mathcal {L}}^\perp \), we have \(\dim ({\mathbb {P}}{\mathcal {L}} \cap H) \ge k-c\). This number exceeds the expected dimension of \({\mathbb {P}}{\mathcal {L}}' \cap H\) among generic \({\mathcal {L}}'\) that contain X. This shows that any subspace \({\mathcal {L}}\) with \(s({\mathcal {L}}) = s\) and \(s({\mathcal {L}}^\perp ) = n-s\) is a point in the coisotropic variety \({\mathrm {Ch}}_{k-c}({\mathcal {X}}_s)\). In particular, a generic element of (9) lies in the following full-dimensional semialgebraic set of real points in \({\mathrm{Ch}}_{k-c}({\mathcal {X}}_s)\):

$$\begin{aligned} \left\{ \,{\mathcal {L}} \in {\mathrm {Gr}}(k, {\mathbb {S}}^n) \mid \exists \, X \in {\mathrm {Reg}}({\mathcal {X}}_s) \cap {\mathbb {S}}^n_+ \cap {\mathcal {L}} \,:\, \dim ({\mathbb {P}}{\mathcal {L}} \cap T_X({\mathcal {X}}_s)) \ge k- c \,\right\} . \end{aligned}$$
(10)

Step 2: If \(0< s({\mathcal {L}}) = r < s\) and \(0 < s({\mathcal {L}}^\perp ) = t \le n-s\). We may assume that our subspace \({\mathcal {L}}\) contains the specific diagonal matrix

$$\begin{aligned} X \,\, = \,\, {\mathrm {diag}}(\underbrace{1, \dots , 1}_{r}, 0, \dots , 0) \,\,\in \,\, {\mathrm {Sing}}({\mathcal {X}}_s). \end{aligned}$$

Here \({\mathrm{Sing}}\) denotes the singular locus. Also, \({\mathcal {L}}^\perp \) contains the diagonal matrix

$$\begin{aligned} Y \,\, = \,\, {\mathrm {diag}}(0, \dots , 0, \underbrace{1, \dots , 1}_t). \end{aligned}$$

For any \(\epsilon > 0\), let \({\mathcal {L}}_\epsilon \) be the span in \({\mathbb {S}}^n\) of \(X^\perp \cap {\mathcal {L}}\) and

$$\begin{aligned} X_\epsilon \,\,=\,\, {\mathrm {diag}}(\underbrace{1, \dots , 1}_r, \underbrace{\epsilon , \dots , \epsilon }_{s-r}, 0, \dots , 0). \end{aligned}$$

Then \(s({\mathcal {L}}_\epsilon ) = s\) and \({\mathcal {L}}_\epsilon \rightarrow {\mathcal {L}}\) as \(\epsilon \rightarrow 0\). Let \(H = T_{X_\epsilon }({\mathcal {X}}_s)\). By construction, we still have \(Y \in {\mathcal {L}}_\epsilon ^\perp \cap H^\perp \). By the proof of Step 1, we conclude that \({\mathcal {L}}_\epsilon \in {\mathrm {Ch}}_{k-c}({\mathcal {X}}_s)\) for all \(\epsilon > 0\). Then \({\mathcal {L}} \in {\mathrm {Ch}}_{k-c}({\mathcal {X}}_s)\) since varieties are closed. In particular, we see that the set of bad instances with \(s({\mathcal {L}}) + s({\mathcal {L}}^\perp ) = n\) is dense in the set of all bad instances, so (9) is Zariski dense in (10).

Next we show that values of s outside the range (6) are covered by those in that range. Suppose that s does not satisfy (6). We claim that the set of bad instances is in the closure of the set of bad subspaces for which s is in the range. Fix an s outside the range (6). Let \({\mathcal {L}}\) be a bad subspace such that either \(s({\mathcal {L}}) = s\) or \(s({\mathcal {L}}^\perp ) = n-s\). There are again two cases.

Case 1: If \(k \le c = \mathrm {codim}({\mathcal {X}}_s)\), then \(k-c \le 0\). Then \({\mathrm{Ch}}_{k-c}({\mathcal {X}}_s)\) is either the Chow form or trivially \({\mathrm{Gr}}(k, {\mathbb {S}}^n)\), neither of which characterizes bad subspaces as argued below. By a similar \(\epsilon \)-argument as that in Step 2, we may assume \(s({\mathcal {L}}) = s\) and \(s({\mathcal {L}}^\perp ) = n-s\). Then \({\mathcal {L}} \cap I_{\mathcal {L}}^2 \subsetneq {\mathcal {L}} \cap I_{\mathcal {L}}\) by Corollary 1. Therefore, if \(X \in {{\mathrm{Reg}}}({\mathcal {X}}_s) \cap {\mathbb {S}}^n_+ \cap {\mathcal {L}}\), then \(\dim ({\mathbb {P}}{\mathcal {L}} \cap T_X({\mathcal {X}}_s)) = \dim {\mathbb {P}}({\mathcal {L}} \cap I_{{\mathcal {L}}}) > \dim {\mathbb {P}}({\mathcal {L}} \cap I_{{\mathcal {L}}}^2) \ge 0\). So, \(\dim ({\mathbb {P}}{\mathcal {L}} \cap T_X({\mathcal {X}}_s)) \ge 1\). The set of bad subspaces is contained in the Zariski closure of the following set whose codimension is greater than 1:

$$\begin{aligned} \left\{ \,{\mathcal {L}} \in {\mathrm {Gr}}(k, {\mathbb {S}}^n) \mid \exists \, X \in {\mathrm {Reg}}({\mathcal {X}}_s) \cap {\mathcal {L}} \,:\, \dim ({\mathbb {P}}{\mathcal {L}} \cap T_X({\mathcal {X}}_s)) \ge 1 \,\right\} . \end{aligned}$$

Fix an integer \(s< s' < n\) such that \(c' = {\mathrm{codim}}({\mathcal {X}}_{s'}) < k \le \dim ({\mathcal {X}}_{n-s'})\). We claim that the set of bad instances in \({\mathrm {Gr}}(k, {\mathbb {S}}^n)\) is contained in \({\mathrm {Ch}}_{k-c'}({\mathcal {X}}_{s'})\) where \(c' = \left( {\begin{array}{c}n-s'+1\\ 2\end{array}}\right) \). Consider the matrices

$$\begin{aligned} X = {\mathrm {diag}}(\underbrace{1, \dots , 1}_s, 0, \dots , 0) \qquad {\mathrm{and}} \qquad X_\epsilon = {\mathrm {diag}}(\underbrace{1, \dots , 1}_s, \underbrace{\epsilon , \dots , \epsilon }_{s'-s}, 0, \dots , 0). \end{aligned}$$

We may assume \(X \in {\mathcal {L}}\) and we define \({\mathcal {L}}_\epsilon \) as above. Thus we have \({\mathcal {L}}_\epsilon \rightarrow {\mathcal {L}}\) as \(\epsilon \rightarrow 0\). Since \(s({\mathcal {L}}_\epsilon ) = s' < n\), we have \(n-s' \ge s({\mathcal {L}}_\epsilon ^\perp ) > 0\) by the paragraph folowing Lemma 1. By the argument in Step 1, we find that \({\mathcal {L}}_\epsilon \in {\mathrm{Ch}}_{k-c'}({\mathcal {X}}_{s'})\) for \(\epsilon > 0\), and therefore \({\mathcal {L}} \in {\mathrm{Ch}}_{k-c'}({\mathcal {X}}_{s'})\).

Case 2: Let \(k > \left( {\begin{array}{c}n+1\\ 2\end{array}}\right) - \left( {\begin{array}{c}s+1\\ 2\end{array}}\right) \). Then \(\left( {\begin{array}{c}n+1\\ 2\end{array}}\right) - k < \left( {\begin{array}{c}s+1\\ 2\end{array}}\right) \), so \({\mathrm{Ch}}_{\left( {\begin{array}{c}n+1\\ 2\end{array}}\right) - k -\left( {\begin{array}{c}s+1\\ 2\end{array}}\right) }({\mathcal {X}}_{n-s})\) is trivially \({\mathrm{Gr}}(\left( {\begin{array}{c}n+1\\ 2\end{array}}\right) - k, {\mathbb {S}}^n)\). This does not characterize the orthogonal complement of bad subspaces. By Corollary 1, bad subspaces are contained in the Zariski closure of the following set whose codimension in the Grassmannian is greater than one:

$$\begin{aligned} \left\{ \,{\mathcal {L}} \in {\mathrm {Gr}}(k, {\mathbb {S}}^n) \mid \exists \, Y \in {\mathrm {Reg}}({\mathcal {X}}_{n-s}) \cap {\mathcal {L}}^\perp \,:\, \dim ({\mathbb {P}}{\mathcal {L}}^\perp \cap T_Y({\mathcal {X}}_{n-s})) \ge 0 \,\right\} . \end{aligned}$$

Choose \(0< s'' < s\) with \({\mathrm{codim}}({\mathcal {X}}_{n-s''}) \le \dim {\mathcal {L}}^\perp < \dim ({\mathcal {X}}_{s''})\). Applying the same \(\epsilon \)-argument as in Case 1, we find \({\mathcal {L}}^\perp \in {\mathrm{Ch}}_{\left( {\begin{array}{c}n+1\\ 2\end{array}}\right) -\left( {\begin{array}{c}s''+1\\ 2\end{array}}\right) - k}({\mathcal {X}}_{n-s''})\). By duality, this means \({\mathcal {L}} \in {\mathrm{Ch}}_{k-c''}({\mathcal {X}}_{s''})\) where \(c'' = {\mathrm{codim}}({\mathcal {X}}_{s''})\). We have now shown that all bad subspaces \({\mathcal {L}}\) with \(s({\mathcal {L}})\) outside the Pataki range lie in one of the coisotropic hypersurfaces \({\mathrm{Ch}}_{k-c}({\mathcal {X}}_s)\) where k satisfies (6).

It remains to be seen that the bad subspaces are Zariski dense in \({\mathrm {Ch}}_{k-c}({\mathcal {X}}_s) \), provided (6) holds. By Step 1, the set of bad subspaces is Zariski dense in (9). Since (9) is Zariski dense in (10), it suffices to show that (10) is Zariski dense in \({\mathrm {Ch}}_{k-c}({\mathcal {X}}_s)\). The subset \({\mathrm {Reg}}({\mathcal {X}}_s) \cap {\mathbb {S}}_n^+\) of positive semidefinite rank s matrices is Zariski dense in the variety \({\mathcal {X}}_s\). The same holds for the incidence variety of pairs \((X, {\mathcal {L}})\), where \({\mathcal {L}}\) is tangent to \({\mathcal {X}}_s\) at X. We project this incidence variety, and its Zariski dense subset given by \(X \in {\mathbb {S}}^n_+\), into the Grassmannian \({\mathrm{Gr}}(k, {\mathbb {S}}^n)\). The image of the latter is Zariski dense in the image of the former. Hence the bad subspaces form a Zariski dense subset of \({\mathrm {Ch}}_{k-c}({\mathcal {X}}_s)\). This completes the proof of Theorem 2. \(\square \)

Remark 1

Our proof gives rise to an explicit parametrization of generic bad subspaces \({\mathcal {L}}\) in \({\mathrm {Ch}}_{k-c}({\mathcal {X}}_s)\) which satisfy \(s({\mathcal {L}}) + s({\mathcal {L}}^\perp ) = n\). We shall present a basis for \({\mathcal {L}}^\perp \), similar to those given for \(k=7,n=4\) in Example 9. Namely, we start with \(q_1 =\sum _{i=s+1}^n x_i^2\). For \(i \ge 2\) we set \(q_i = q_i' + \sum _{j=1}^s \ell _{ij} x_j\), where \(q_i'\) is a generic quadratic form in \(x_1, \dots , x_s\) of trace zero, and the \(\ell _{ij}\) are linear forms in \(x_{s+1}, \dots , x_n\). The dimension of \({\mathcal {L}}^\perp \) is supposed to be \(\left( {\begin{array}{c}n+1\\ 2\end{array}}\right) - k\). If k is within the range (6) then \(\dim ({\mathcal {L}}^\perp ) < \left( {\begin{array}{c}n+1\\ 2\end{array}}\right) - \left( {\begin{array}{c}n-s+1\\ 2\end{array}}\right) = \left( {\begin{array}{c}s+1\\ 2\end{array}}\right) + s(n-s)\). Hence, there is enough freedom to keep all \(q_1,q_2,q_3,\ldots \) linearly independent. The resulting subspaces \({\mathcal {L}}\) are generically bad, and they form the Zariski dense subset (9) of \({\mathrm {Ch}}_{k-c}({\mathcal {X}}_s) \).

Our main result, Theorem 2, identifies the subvarieties in the Grassmannian that are responsible for bad behavior in semidefinite programming (SDP). However, these are complex projective varieties and hence they are one step removed from the real figures that are of interest in optimization theory. We close this section by returning to the real and semidefinite setting. The argument in the last paragraph in the proof of Theorem 2 gives rise to the following corollary, aimed at capturing in precise terms what the typical bad subspaces are.

Corollary 2

Fix integers n, k, s as in(6). A generic real subspace \({\mathcal {L}}\) in the component \({\mathrm{Ch}}_{k-c}({\mathcal {X}}_s) \) of the bad variety \( {\mathrm{Bad}}_{k,n}\) is tangent to \({\mathcal {X}}_s\) at a unique matrix \(X \in {\mathbb {S}}^n\), and this X is real. A generic space \({\mathcal {L}}\) such that the matrix X is positive semidefinite is bad.

4 Algebraic computations

In this section we develop computational tools for the geometric problem studied in this paper. Suppose we are given matrices \(A_1,A_2,\ldots ,A_k\) in \({\mathbb {S}}^n\) whose entries are rational numbers. The most basic decision problem is to determine whether or not \({\mathcal {L}} = {\mathbb {R}}\{A_1,A_2,\ldots ,A_k\}\) is a bad subspace, and to find a certificate as in Theorem  1. The first step in this decision process is the computation of the spectrahedral rank. Recall that \(s({\mathcal {L}})\) is the largest rank of any matrix X in the spectrahedral cone \({\mathcal {L}} \cap {\mathbb {S}}^n_+\). For a generic instance \({\mathcal {L}}\), we have \(s({\mathcal {L}}) = 0\) or \(s({\mathcal {L}}) = n\). These are the easy cases, where the image \(\pi _{\mathcal {L}}({\mathbb {S}}^n_+)\) is either all of \({\mathbb {R}}^k\) or a closed pointed cone in \({\mathbb {R}}^k\). We are interested in the decision boundary between these two regimes.

An upper bound on \(s({\mathcal {L}})\) is given by the rank of any matrix Y in \({\mathcal {L}}^\perp \cap {\mathbb {S}}^n_+\). Indeed, we have \(s({\mathcal {L}}) + s({\mathcal {L}}^\perp ) \le n \), with equality for all good subspaces \({\mathcal {L}}\), and for generic bad ones. This leads us to consider the following system of polynomial equations in \(2 \left( {\begin{array}{c}n+1\\ 2\end{array}}\right) \) unknowns, which is derived in [13, Corollary 12.12].

$$\begin{aligned} X \in {\mathcal {L}} \quad {\mathrm{and}} \quad Y \in {\mathcal {L}}^\perp \quad {\mathrm{and}} \quad X\cdot Y = 0 . \end{aligned}$$
(11)

The pair (X, Y) represents a point in the product space \({\mathbb {P}}{\mathcal {L}} \times {\mathbb {P}}{\mathcal {L}}^\perp \) inside \({\mathbb {P}}({\mathbb {S}}^n) \times {\mathbb {P}}({\mathbb {S}}^n)\). We call (11) the critical equations, and its solution set is the critical variety of the subspace \({\mathcal {L}}\). See [13, Sect. 12.2] for a textbook introduction. As is customary in algebraic geometry, we work in the complex projective setting, with \(\,{\mathbb {P}}{\mathcal {L}} \simeq {\mathbb {P}}^{k-1}\,\) and \(\,{\mathbb{P}}{{\mathcal{L}}^{\perp} } \simeq {\mathbb{P}}^{\left( {\begin{array}{l}n+1\\ 2\end{array}}\right) -k-1}\). Thus \({\mathbb {P}}{\mathcal {L}} \times {\mathbb {P}}{\mathcal {L}}^\perp \) is a variety of dimension \(\left( {\begin{array}{c}n+1\\ 2\end{array}}\right) -2\).

The Eq. (11) are reminiscent of the optimality conditions for SDP, in the notation used in [2, (5)], [13, (12.14)] and [14, (3.4)]. However, there is a crucial distinction. The optimality conditions are based on a flag of subspaces \({\mathcal {V}} \subset {\mathcal {U}}\) where \({\mathrm{dim}}({\mathcal {U}}) - {\mathrm{dim}}({\mathcal {V}}) = 2\). They have many solutions in \({\mathbb {P}}({\mathbb {S}}^n) \times {\mathbb {P}}({\mathbb {S}}^n) \), counted by the algebraic degree of SDP, as shown in [14]. In our setting, the flag \({\mathcal {V}} \subset {\mathcal {U}}\) is replaced by \({\mathcal {L}} \subseteq {\mathcal {L}}\). The Eq. (11) are expected to have no solutions. The critical variety of a generic subspace \({\mathcal {L}}\) is the empty set in \({\mathbb {P}}({\mathbb {S}}^n) \times {\mathbb {P}}({\mathbb {S}}^n)\). What we care about are the exceptional \({\mathcal {L}}\) for which (11) has a solution.

We first review the meaning of the equations \(X\cdot Y = 0\). These represent complementary slackness in SDP. The normal cycle of the PSD cone is the semialgebraic set

$$\begin{aligned} {\mathrm{NC}}_n \,\, = \,\, \left\{ (X,Y) \in ({\mathbb {S}}^n_+)^2 \,: \, X \cdot Y = 0 \right\} . \end{aligned}$$
(12)

The normal cycle represents pairs of points in the cone together with supporting hyperplanes. In the real affine version seen in (12), this is a semialgebraic set of middle dimension \(\left( {\begin{array}{c}n+1\\ 2\end{array}}\right) \). If X ranges over matrices of rank s then Y ranges over complementary matrices of rank \(n-s\). For an algebraic geometer, it is more natural to consider the complex projective version. This is a reducible complex algebraic variety, here referred to as the projective normal cycle:

$$\begin{aligned} {{\mathrm{PNC}}}_n \,\, = \,\, \left\{ (X,Y) \in ({\mathbb {P}}({\mathbb {S}}^n))^2 \,: \, X \cdot Y = 0 \right\} . \end{aligned}$$
(13)

Proposition 2

The projective normal cycle \({\mathrm{PNC}}_n\) has \(n-1\) irreducible components, each of dimension \(\left( {\begin{array}{c}n+1\\ 2\end{array}}\right) -2\). These are the conormal varieties of the varieties \({\mathcal {X}}_s\).

Proof

This is the content of [7, Proposition I.4.11], revisited for SDP in [14, Proposition 12] and [1, Example 5.15]. The dimension statement is [1, Proposition 5.10]. \(\square \)

Fix a generic point \({\mathcal {L}}\) in the Grassmannian \({\mathrm{Gr}}(k,{\mathbb {S}}^n)\). For dimension reasons, the intersection \(\,({\mathbb {P}}{\mathcal {L}} \times {\mathbb {P}}{\mathcal {L}}^\perp ) \,\cap \,{\mathrm{PNC}}_n\,\) defined by (11) will be the empty set in \(({\mathbb {P}}({\mathbb {S}}^n))^2\). We are interested in subspaces \({\mathcal {L}}\) for which that intersection is nonempty.

Theorem 3

The set \({\mathcal {L}}\) such (11) has a solution (X, Y) contains the bad variety:

$$\begin{aligned} \left\{ \,{\mathcal {L}} \in {\mathrm{Gr}}(k, {\mathbb {S}}^n)\,:\, \,({\mathbb {P}}{\mathcal {L}} \times {\mathbb {P}}{\mathcal {L}}^\perp ) \,\cap \,{\mathrm{PNC}}_n \,\not = \, \emptyset \,\right\} \,\, \supseteq \,\,{\mathrm{Bad}}_{k,n}. \end{aligned}$$
(14)

Equality holds unless \(k=\left( {\begin{array}{c}n-s+1\\ 2\end{array}}\right) \), when the difference is the Chow form \({\mathrm{Ch}}_0({\mathcal {X}}_s)\). The closure of the set of bad subspaces is the set of real \({\mathcal {L}}\) for which \({\mathcal {L}} \times {\mathcal {L}}^\perp \) intersects the normal cycle \({\mathrm{NC}}_n\) nontrivially, i.e. (11) has a solution (X, Y) with \(X {\not =} \,0,Y {\not =} \,0\).

Proof

This follows from Theorem 2 and Corollary 2, using the fact about conormal varieties stated in Proposition 2. It suffices to show that a Zariski dense subset of the bad variety \({\mathrm{Bad}}_{k, n}\) is contained in the left hand side of (14). We choose that subset to be the union of (10) over all s within the range (6). Fix s and consider \({\mathcal {L}}\) in (10). Pick \(X \in {\mathrm{Reg}}({\mathcal {X}}_s) \cap {\mathcal {L}}\) such that \(\dim ({\mathbb {P}}{\mathcal {L}} \cap T_X({\mathcal {X}}_s)) \ge k-c\). Let \(I_{{\mathcal {L}}}\) and \(I_{{\mathcal {L}}^\perp }\) be defined as in the paragraph prior to Corollary 1. Let \({\mathcal {L}}'\) be the image of \({\mathcal {L}}\) under the map \({\mathbb {S}}^n \rightarrow {\mathbb {S}}^{n-s}\) which restricts the domain of each matrix to the variety of \(I_{{\mathcal {L}}}\). Then \(\dim ({\mathcal {L}}' )\le c-1 = \dim ({\mathbb {S}}^{n-s}) -1\), so that \(\pi _{{\mathcal {L}}'}: {\mathbb {S}}^{n-s} \rightarrow {\mathcal {L}}'^\vee \) has a nontrivial kernel. If \(Y \in \ker (\pi _{{\mathcal {L}}'})\), then \(Y \in {\mathcal {L}}^\perp \) and \(X \cdot Y = 0\).

The same relationship between coisotropic hypersurfaces and the conormal variety extends from our specific varieties \({\mathcal {X}}_s\) to arbitrary projective varieties. This is essentially biduality, and we view it as a geometric refinement of [11, Sect. 4]. \(\square \)

Remark 2

The “unless” statement in the second sentence of Theorem 3 looks mysterious at first sight. We here offer an explanation for the case \(n=k=3\) and \(s=1\). The left hand side of (14) equals \({\mathrm{Ch}}_2({\mathcal {X}}_2) \,\cup \, {\mathrm{Ch}}_0({\mathcal {X}}_1)\). We now derive the irreducible polynomials for the two components. Following [13, Example 5.3], we introduce nine affine coordinates \(a,b,\ldots ,h,i\) on \({\mathrm{Gr}}(3,{\mathbb {S}}^3)\). To this end, we fix bases \(\{q_1,q_2,q_3\}\) for \({\mathcal {L}}\) and \(\{r_1,r_2,r_3\}\) for \({\mathcal {L}}^\perp \) as follows:

$$\begin{aligned} \begin{matrix} q_1 = x_1^2+ax_1 x_2+b x_1 x_3+c x_2 x_3\,, &{} \quad &{} r_1 = a x_1^2+d x_2^2+g x_3^2-2 x_1 x_2,\\ q_2 = x_2^2+dx_1 x_2+e x_1 x_3+ f x_2 x_3\,, &{} \quad &{} r_2 = b x_1^2+e x_2^2+h x_3^2-2 x_1 x_3, \\ q_3 = x_3^2+g x_1 x_2+h x_1 x_3+i x_2 x_3\,, &{} \quad &{} r_3 = c x_1^2+f x_2^2+i x_3^2 - 2 x_2 x_3. \\ \end{matrix} \end{aligned}$$

The coisotropic hypersurface \({\mathrm{Ch}}_2({\mathcal {X}}_2)\) is defined by a resultant with 218 terms:

$$\begin{aligned} \begin{matrix} {\mathrm{Res}}(q_1,q_2,q_3) &{} \! = \!\! &{} a^2 b d e f h i^2-a^2 b d f^2 h^2 i-a^2 b e^2 f g i^2+a^2 b e f^2 g h i-a^2 c d e^2 h i^2+a^2 c d e f h^2 i \\ &{} &{} +\,a^2 c e^3 g i^2- \cdots +a f h+b d i+b f g+c d h+3 c e g-2 a d-2 b h-2 f i+1. \end{matrix} \end{aligned}$$

The coisotropic hypersurface \({\mathrm{Ch}}_0({\mathcal {X}}_1)\) is defined by a resultant with 549 terms:

$$\begin{aligned} \begin{matrix} {\mathrm{Res}}(r_1,r_2,r_3) &{} = &{} a^4 e^4 i^4-4 a^4 e^3 f h i^3+6 a^4 e^2 f^2 h^2 i^2-4 a^4 e f^3 h^3 i+a^4 f^4 h^4 -4 a^3 b d e^3 i^4\\ &{} &{} \!\! +\,12 a^3 b d e^2 f h i^3 - \cdots +384 c e f g i -128 c f^2 g h+256 a b e g+256 c e h i-512 c e g. \end{matrix} \end{aligned}$$

The bad variety \({\mathrm{Bad}}_{3,3}\) equals \({\mathrm{Ch}}_2({\mathcal {X}}_2)\). All generic subspaces in \({\mathrm{Ch}}_0({\mathcal {X}}_1)\) are good.

The objects in (11), (12) and (13) are symmetric under switching the two factors. But this is not the case when it comes to \(\pi _{\mathcal {L}}({\mathbb {S}}^n_+)\) being closed. For deciding between bad and good, the symmetry between primal and dual is broken. From an SDP perspective, one can see this in Proposition 1 of [17, Sect. 3]. For a concrete example, consider the dual pair \({\mathcal {L}} = {\mathbb {R}}\{x_1^2+x_2^2,x_1 x_2+x_1 x_3,x_2 x_3 \}\) and \({\mathcal {L}}^\perp = {\mathbb {R}}\{x_3^2,x_1^2-x_2^2,x_1x_2-x_1x_3 \}\). Then \({\mathcal {L}} \in {\mathrm{Ch}}_2({\mathcal {X}}_2) \backslash {\mathrm{Ch}}_0({\mathcal {X}}_1)\) is bad, and quite typical for this, whereas \({\mathcal {L}}^\perp \in {\mathrm{Ch}}_1({\mathcal {X}}_0) \backslash {\mathrm{Ch}}_2({\mathcal {X}}_2)\) is good.

We can use Theorem 3 to compute equations that define our coisotropic hypersurfaces. Namely, consider the incidence variety in \({\mathbb {P}}({\mathbb {S}}^n) \times {\mathbb {P}}({\mathbb {S}}^n) \times {\mathrm{Gr}}(k,{\mathbb {S}}^n)\) that is defined by the critical equations (11). The hypersurface we are interested in is the image of that incidence variety under the map \(\,{\mathbb {P}}({\mathbb {S}}^n) \times {\mathbb {P}}({\mathbb {S}}^n) \times {\mathrm{Gr}}(k,{\mathbb {S}}^n) \rightarrow {\mathrm{Gr}}(k,{\mathbb {S}}^n)\). In particular, if we fix some particular rank s, then the image of the incidence variety in \(\,{\mathcal {X}}_s \times {\mathcal {X}}_{n-s} \times {\mathrm{Gr}}(k,{\mathbb {S}}^n)\) is the irreducible hypersurface \({\mathrm{Ch}}_{k-c}({\mathcal {X}}_s)\) in \( {\mathrm{Gr}}(k,{\mathbb {S}}^n)\). Algebraically, one obtains the polynomial defining \({\mathrm{Ch}}_{k-c}({\mathcal {X}}_s)\) by eliminating X and Y from the following rank s critical equations of \({\mathcal {L}}\):

$$\begin{aligned} X \in {\mathcal {L}} \,\,\, {\mathrm{and}} \,\,\, Y \in {\mathcal {L}}^\perp \,\,\, {\mathrm{and}} \,\,\, X\cdot Y = 0 \,\,\, {\mathrm{and}} \,\,\, \mathrm{rank}(X) \le s \,\,\, {\mathrm{and}} \,\,\, \mathrm{rank}(Y) \le n-s. \end{aligned}$$
(15)

Corollary 3

Under the assumption of Corollary 2, the system (15) has a unique solution (X, Y) in \(({\mathbb {P}}({\mathbb {S}}^n))^2\). Here X is the point of tangency and Y is in the normal space.

Example 10

This Macaulay2 code represents the system (15) for \(n=k=2, s=1\):

figure a

The following eliminates the pair (X, Y) and retains the subspace \({\mathcal {L}} = \mathrm{span}(\mathtt{A},\mathtt{B})\).

figure b

As predicted, the output is the resultant \(R = {\mathrm{Bad}}_{2,2} = {\mathrm{Ch}}_1({\mathcal {X}}_1)\) from Example 2. Note the importance of the saturation step in representing subschemes of \(({\mathbb {P}}({\mathbb {S}}^n))^2\).

Example 11

(\(n=3,k=4,s=1\)) We represent \({\mathcal {L}}^\perp \) by a basis of \(3 \times 3\)-matrices \(\mathtt{U}\) and \(\mathtt{V}\). Their \(6+6\) entries are the dual Stiefel coordinates on \( {\mathrm{Gr}}(4,{\mathbb {S}}^3)\). We run

figure c

The output is a polynomial in u1..u6,v1..v6 with 3210 terms of degree 12. This is the tact invariant we saw in Example 8. It defines \({\mathrm{Bad}}_{4,3} = {\mathrm{Ch}}_1({\mathcal {X}}_1) \simeq {\mathrm{Ch}}_1({\mathcal {X}}_2)\).

We close with two case studies on numerical examples. In both cases, the critical variety defined by (15) consists of a single rational point, and it is computed using the command criticalIdeal in the Macaulay2 package SemidefiniteProgramming [3]. By using this package, we can compare the algebraic approach described above with Pataki’s facial reduction [15] in the usual numerical framework of SDP.

Example 12

(\(n=k=4\)) Let t be a parameter and \({\mathcal {L}}_t\) the space with basis

$$\begin{aligned} \! A_1 = \begin{bmatrix} 180 \! &{} \! 112 \! &{} \! 205 \! &{} \! 131\\ 112 \! &{} \! 88 \! &{} \! 131 \! &{} \! 96 \\ 205 \! &{} \! 131 \! &{} \! 228 \! &{} \! 152 \\ 131 \! &{} \! 96 \! &{} \! 152 \! &{} \! 104 \end{bmatrix} \! , \, A_2 = \begin{bmatrix} 428 \! &{} \! 253 \! &{} \! 473 \! &{} \! 288 \\ 253 \! &{} \! 238 \! &{} \! 262 \! &{} \! 227 \\ 473 \! &{} \! 262 \! &{} \! 516 \! &{} \! 307 \\ 288 \! &{} \! 227 \! &{} \! 307 \! &{} \! 168 \\ \end{bmatrix} \! , \, A_3 = \begin{bmatrix} 216 \! &{} \! 123 \! &{} \! 234 \! &{} \! 137 \\ 123 \! &{} \! 128 \! &{} \! 118 \! &{} \! 116 \\ 234 \! &{} \! 118 \! &{} \! 252 \! &{} \! 138 \\ 137 \! &{} \! 116 \! &{} \! 138 \! &{} \! 68 \end{bmatrix} \!, \, A_4 = \begin{bmatrix} 320 \!&{} \! t \!&{}\! 380 \!&{}\! 254 \\ t \! &{} \! 140 \! &{} \! 258 \! &{} \! 166 \\ 380 \! &{} \! 258 \! &{} \! 448 \! &{} \! 342 \\ 254 \! &{} \! 166 \! &{} \! 342 \! &{} \! 208 \end{bmatrix}\! . \end{aligned}$$

Is there a value of t for which \({\mathcal {L}}_t\) is bad? Geometrically, \(\{{\mathcal {L}}_t\}_{t \in {\mathbb {R}}}\) is a line in \({\mathrm{Gr}}(4,{\mathbb {S}}^4) \subset {\mathbb {P}}^{209}\). What is its intersection with \({\mathrm{Bad}}_{4,4}\)? To answer this question, we consider \(s=2\) and we evaluate the Hurwitz form \({\mathrm{Ch}}_1({\mathcal {X}}_2)\) on \({\mathcal {L}}_t\); see [21]. The result is a primitive polynomial \(f \in {\mathbb {Z}}[t]\) of degree 30. One of its roots is 194. Each coefficient has over 100 digits. The leading coefficient equals 34006196837917896573795931 713719797442228459580467476929073669732608826214924094017413181247522771810155185.

We now substitute \(t=194\), and we continue our computation in the polynomial ring \({\mathbb {Q}}[x_1,x_2,x_3,x_4,y_{11}, y_{12}, \ldots ,y_{44}]\). We write \(X = x_1 A_1 + x_2 A_2 + x_3 X_3 + x_4 X_4\) for a general matrix in \({\mathcal {L}}\). The constraint \(Y \in {\mathcal {L}}^\perp \) is encoded by the four linear equations \( A_i \circ Y = 0\) in the ten entries of the matrix \(Y = (y_{ij})\). The ideal for (15) is given by the \(3\times 3\)-minors of X and Y as well as the entries of \(X\cdot Y \). After saturating by the irrelevant ideal of \({\mathbb {P}}{\mathcal {L}} \times {\mathbb {P}}({\mathbb {S}}^4)\), we obtain the homogeneous maximal ideal of the unique point \((X^*,Y^*)\) in the critical variety:

$$\begin{aligned} {\begin{matrix} \langle \, x_1-14x_4, x_2+18x_4, x_3-24x_4\,, \,\, 314 y_{11} -197 y_{44} ,314 y_{12}-11 y_{44}, 314 y_{12}-11 y_{44}, 314y_{13}+213 y_{44}, \\ \,\,\,\,314 y_{14}-53 y_{44},157 y_{22}-313 y_{44}, 157 y_{23} +30 y_{44},157 y_{24} +215 y_{44}, 157 y_{33}-117 y_{44}, 157 y_{34} +12 y_{44} \, \rangle . \end{matrix}} \end{aligned}$$

The matrices \(X^*\) and \(Y^*\) are unique up to scaling. They are positive semidefinite of rank 2. For instance, \(X^* = 14 A_1 - 18 A_2 + 24 A_3 + A_4\) is the point of tangency for \({\mathcal {L}}\) at the variety \({\mathcal {X}}_2\). For an alternative view, we consider the degenerate SDP

$$\begin{aligned} {\mathrm{Minimize}}\,\, 0 \circ Y \quad \hbox {subject to} \,\,\,\, A_1 \circ Y = A_2 \circ Y =A_3 \circ Y =A_4 \circ Y = 0 . \end{aligned}$$
(16)

We can solve this numerically in Macaulay2, using the following commands:

figure d

The numerical output approximates the point \((X^*,Y^*)\) in the critical variety of \({\mathcal {L}}\).

Example 13

(\(n= 4, k = 5, s=2 \)) Let \(A_1,\ldots ,A_5\) be the five matrices displayed in equation (2.1) of [17, Example 4]. Their linear span \({\mathcal {L}} \in {\mathrm{Gr}}(5,{\mathbb {S}}^4) \) is a bad subspace. However, \({\mathcal {L}}\) is not generic in the sense of Corollary 2. To see this, we compute the saturation of the ideal specified in (15). We find that the critical variety is a reducible surface in \({\mathbb {P}}{\mathcal {L}} \times {\mathbb {P}}{\mathcal {L}}^\perp \). We conclude that \({\mathcal {L}}\) is a point in \({\mathrm{Bad}}_{5,4}\), but it does not satisfy the hypothesis in Corollary 3, as that would imply that the variety is only one point. The projection of the critical variety into \({\mathbb {P}}{\mathcal {L}} \simeq {\mathbb {P}}^4\) is the plane defined by \(\langle 3 x_1+ 2 x_3+3 x_4+9 x_5, 3 x_2-x_3+3 x_5 \rangle \). The point \(x = (-1,-1,0,-2,1)\) lies in that plane. It specifies the matrix \(X = \sum _{i=1}^5 x_i A_i = \mathrm{diag}(1,1,0,0)\), seen on the right in [17, Eq. (2.4)]. Note that X is in \( {\mathcal {L}} \cap {\mathbb {S}}^4_+\) and \(\mathrm{rank}(X) = s({\mathcal {L}})=2\).

The bad variety \({\mathrm{Bad}}_{5,4}\) is the hypersurface \( {\mathrm{Ch}}_2({\mathcal {X}}_2)\), which is self-dual and has degree \(\delta (5,4,2) = 42\). To construct typical points, we choose random \(B_1,B_2,B_3\) in \({\mathbb {S}}^4\), and we replace \(A_1\) by \(B_1\) and \(A_5\) by \(B_2 + t B_3\), where t is a new unknown. Let \({\mathcal {L}}_t\) denote the resulting subspace. We repeat the above computation of (15) for \({\mathcal {L}}_t\). By eliminating all 15 variables \(x_i\) and \( y_{jk}\), we obtain a principal ideal \(\langle f(t) \rangle \), where \(f \in {\mathbb {Z}}[t]\) has degree 42. This is the restriction of \({\mathrm{Ch}}_2({\mathcal {X}}_2)\) to the line \(\{ {\mathcal {L}}_t \}_{t \in {\mathbb {C}}}\) in \( {\mathrm{Gr}}(5,{\mathbb {S}}^4) \subset {\mathbb {P}}^{251}\). The real roots of f(t) are the candidates for bad subspaces \({\mathcal {L}}_t\).

We experimented with the degenerate SDP given by \(A_1,A_2,A_3,A_4,A_5\). Using SemidefiniteProgramming in Macaulay2 [3, 8], we found that the critical ideal has codimension 11 and degree 11. It was faster to compute the projection of the critical variety into \({\mathbb {P}}{\mathcal {L}} \simeq {\mathbb {P}}^4\) after the change of coordinates in [17, (2.3)]. However, the numerical solver performed better before the change of coordinates, and it gave

figure e

This is the primal-dual pair \((X^*,Y^*)\) of numerical solutions to our critical Eq. (15). Rounding small numbers down to zero, we see that approximately \(\mathrm{rank} (X^*) = 2\) while \(\mathrm{rank}(Y^*) = 1\). This is consistent with theoretical analysis. If the subspace \({\mathcal {L}}\) were generic in \({\mathrm{Bad}}_{5,4}\), then the rank of these matrices add up to \(n=4\). They would be unique up to scaling and their entries would be algebraic numbers of degree \(42 = \delta (5,4,2)\) over the rationals \({\mathbb {Q}}\).