A Sobolev-Type Inequality for the Curl Operator and Ground States for the Curl–Curl Equation with Critical Sobolev Exponent

Abstract

Let $\Omega \subset \mathbb {R}^3$ be a Lipschitz domain and let $S_\mathrm {curl}(\Omega )$ be the largest constant such that

$$\begin{aligned} \int _{\mathbb {R}^3}|\nabla \times u|^2\, \mathrm{d}x\ge S_{\mathrm {curl}}(\Omega ) \inf _{\begin{array}{c} w\in W_0^6(\mathrm {curl};\mathbb {R}^3)\\ \nabla \times w=0 \end{array}}\Big (\int _{\mathbb {R}^3}|u+w|^6\,\mathrm{d}x\Big )^{\frac{1}{3}} \end{aligned}$$

for any u in $W_0^6(\mathrm {curl};\Omega )\subset W_0^6(\mathrm {curl};\mathbb {R}^3)$, where $W_0^6(\mathrm {curl};\Omega )$ is the closure of $\mathcal {C}_0^{\infty }(\Omega ,\mathbb {R}^3)$ in $\{u\in L^6(\Omega ,\mathbb {R}^3): \nabla \times u\in L^2(\Omega ,\mathbb {R}^3)\}$ with respect to the norm $(|u|_6^2+|\nabla \times u|_2^2)^{1/2}$. We show that $S_{\mathrm {curl}}(\Omega )$ is strictly larger than the classical Sobolev constant S in $\mathbb {R}^3$. Moreover, $S_{\mathrm {curl}}(\Omega )$ is independent of $\Omega$ and is attained by a ground state solution to the curl–curl problem

$$\begin{aligned} \nabla \times (\nabla \times u) = |u|^4u \end{aligned}$$

if $\Omega =\mathbb {R}^3$. With the aid of these results we also investigate ground states of the Brezis–Nirenberg-type problem for the curl–curl operator in a bounded domain $\Omega$

$$\begin{aligned} \nabla \times (\nabla \times u) +\lambda u = |u|^4u\quad \hbox {in }\Omega , \end{aligned}$$

with the so-called metallic boundary condition $\nu \times u=0$ on $\partial \Omega$, where $\nu$ is the exterior normal to $\partial \Omega$.

1 Introduction

Sobolev-type inequalities have been widely studied by a large number of authors and the best Sobolev constants play an important role in a variety of fields, such as the theory of partial differential equations, differential geometry, isoperimetric inequalities as well as in mathematical physics; see for example [4, 20, 33]. In particular, if $\Omega$ is a domain in $\mathbb {R}^3$, then the best constant S in the Sobolev inequality

$$\begin{aligned} \int _{\Omega }|\nabla u|^2\,\mathrm{d}x\ge S \Big (\int _{\Omega }|u|^6\,\mathrm{d}x\Big )^{\frac{1}{3}}\quad \hbox { for }u\in {{\mathcal {D}}}^{1,2}(\Omega ) \end{aligned}$$

(1.1)

has been computed explicitly by Talenti [33] and as is well-known, it is achieved (that is, equality holds) if and only if $\Omega =\mathbb {R}^3$ and u is the Aubin–Talenti instanton $U_{\varepsilon ,y}(x) {:=} 3^{1/4}(\varepsilon ^2+|x-y|^2)^{-1/2}$, see [4, 33]. When $\varepsilon =1$, this is the unique (up to translations in $\mathbb {R}^3$) positive solution to the equation $-\Delta u=|u|^4u$ in ${{\mathcal {D}}}^{1,2}(\mathbb {R}^3)$ and a ground state, that is, a minimizer for the energy functional among all nontrivial solutions.

The aim of this work is to perform a similar analysis for the curl operator $\nabla \times (\cdot )$. This is challenging from the mathematical point of view and important in mathematical physics; such operator appears for example in Maxwell equations as well as in Navier–Stokes problems [13, 17, 26]. Finding a formulation in the spirit of (1.1), but involving the curl operator, is not straightforward and there are several essential difficulties as we shall see later.

For instance, the kernel of $\nabla \times (\cdot )$ is of infinite dimension since $\nabla \times (\nabla \varphi )=0$ for all $\varphi \in {{\mathcal {C}}}^2(\Omega )$. Hence the inequality (1.1) with $\nabla u$ replaced by $\nabla \times u$ would hold for all $u\in {{\mathcal {C}}}_0^\infty (\mathbb {R}^3,\mathbb {R}^3)$ only if $S=0$. This makes it necessary to introduce a Sobolev-like constant in a different way which we now proceed to do.

Let $\Omega$ be a Lipschitz domain in $\mathbb {R}^3$ and for $2\le p\le 6$, let

$$\begin{aligned} W^p(\mathrm {curl};\Omega ) {:=} \{u\in L^p(\Omega ,\mathbb {R}^3): \nabla \times u\in L^2(\Omega ,\mathbb {R}^3)\}. \end{aligned}$$

This is a Banach space if provided with the norm

$$\begin{aligned} \Vert u\Vert _{W^p(\mathrm {curl};\Omega )} {:=} \left( |u|^2_p+|\nabla \times u|^2_2\right) ^{1/2}. \end{aligned}$$

Here and in the sequel $|\cdot |_q$ denotes the $L^q$-norm for $q\in [1,\infty ]$. We also define

$$\begin{aligned} W^p_0(\mathrm {curl};\Omega ) {:=} \text {closure of } {{\mathcal {C}}}^{\infty }_0(\Omega ,\mathbb {R}^3) \text { in }W^p(\mathrm {curl};\Omega ). \end{aligned}$$

(1.2)

If $\Omega =\mathbb {R}^3$, these two spaces coincide, see Lemma 2.1. Although results of this kind are well known, we provide a proof for the reader’s convenience. The spaces $W^2(\mathrm {curl};\Omega )$ and $W^2_0(\mathrm {curl};\Omega )$ are studied in detail in [13, 18, 26]. Extending $u\in W^p_0(\mathrm {curl};\Omega )$ by 0 outside $\Omega$ we may assume $W^p_0(\mathrm {curl};\Omega )\subset W^p_0(\mathrm {curl};\mathbb {R}^3)$. Denote the kernel of $\nabla \times (\cdot )$ in $W^6_0(\mathrm {curl};\mathbb {R}^3)$ by

$$\begin{aligned} {{\mathcal {W}}}{:=}\{w\in W^6_0(\mathrm {curl};\mathbb {R}^3): \nabla \times w=0\}. \end{aligned}$$

Let $S_{\mathrm {curl}}(\Omega )$ be the largest possible constant such that the inequality

$$\begin{aligned} \int _{\mathbb {R}^3}|\nabla \times u|^2\, \mathrm{d}x\ge S_{\mathrm {curl}}(\Omega )\inf _{w\in {{\mathcal {W}}}}\Big (\int _{\mathbb {R}^3}|u+w|^6\,\mathrm{d}x\Big )^{\frac{1}{3}} \end{aligned}$$

(1.3)

holds for any $u\in W^6_0(\mathrm {curl};\Omega ){\setminus } \mathcal {W}$. Inequality (1.3) is in fact (trivially) satisfied also for $u\in W^6_0(\mathrm {curl};\Omega )\cap {{\mathcal {W}}}$ because then both sides are zero. Note that here u but not necessarily w is supported in $\Omega$. It is not a priori clear that $S_{\mathrm {curl}}(\Omega )$ is positive or that it is independent of $\Omega$. That this is the case follows from Theorems 1.1 and 1.2(a) below.

Theorem 1.1

$S_{\mathrm {curl}}(\Omega ) = S_\mathrm {curl}$ where $S_\mathrm {curl}{:=} S_\mathrm {curl}(\mathbb {R}^3)$.

In the next result we show that $S_{\mathrm {curl}}$ is attained provided $\Omega =\mathbb {R}^3$ and the optimal function is (up to rescaling) a ground state solution to the curl–curl problem with critical exponent. Existence of a ground state in this case has been an open question for some time. Let

$$\begin{aligned} J(u){:=}\frac{1}{2}\int _{\mathbb {R}^3} |\nabla \times u|^2\,\mathrm{d}x- \frac{1}{6}\int _{\mathbb {R}^3} |u|^6\, \mathrm{d}x \end{aligned}$$

(1.4)

and introduce the following constraint:

$$\begin{aligned} {{\mathcal {N}}}{:=}\Big \{u\in W^6_0(\mathrm {curl};\mathbb {R}^3){\setminus }{{\mathcal {W}}}: \int _{\mathbb {R}^3}|\nabla \times u|^2=\int _{\mathbb {R}^3}|u|^6\, \mathrm{d}x\hbox { and }\mathop {\mathrm {div}}(|u|^4u)=0\Big \}.\nonumber \\ \end{aligned}$$

(1.5)

As we shall see later, this set is a variant of a generalization of the Nehari manifold [27] which may be found in [28] for a Schrödinger equation.

Theorem 1.2

• (a) $S_\mathrm {curl}> S$.

• (b) $\inf _{{{\mathcal {N}}}}J=\frac{1}{3}S_{\mathrm {curl}}^{3/2}$ and is attained. Moreover, if $u\in {{\mathcal {N}}}$ and $J(u)=\inf _{{{\mathcal {N}}}} J$, then u is a ground state solution to the equation

  $$\begin{aligned} \nabla \times (\nabla \times u)= |u|^4u\quad \hbox {in }\mathbb {R}^3 \end{aligned}$$

  (1.6)

  and equality holds in (1.3) for this u. If u satisfies equality in (1.3), then there are unique $t>0$ and $w\in {{\mathcal {W}}}$ such that $t(u+w)\in {{\mathcal {N}}}$ and $J(t(u+w))=\inf _{{{\mathcal {N}}}} J$.

A natural question arises whether ground states must have some symmetry properties. It follows from Theorem 1.1 in [5] that any ${{\mathcal {O}}}(3)$-equivariant (weak) solution to (1.6) is trivial, hence a ground state cannot be radially symmetric.

The curl–curl problem $\nabla \times (\nabla \times u)= f(x,u)$ in a bounded domain or in $\mathbb {R}^3$ has been recently studied for example in [5,6,7,8, 22, 24] under different hypotheses on f but always assuming f is subcritical, that is, $f(x,u)/|u|^5\rightarrow 0$ as $|u|\rightarrow \infty$. However, the occurence of ground states to (1.6) (that is, in the critical exponent case) has been an open problem as we have already mentioned. In view of the existence of Aubin–Talenti instantons, this is a very natural question. While the instantons are given explicitly, we have no such explicit formula for ground states in the curl–curl case. Since the instantons are radially symmetric up to translations, one can find them by ODE methods. In view of the above remark concerning ${{\mathcal {O}}}(3)$-equivariant solutions, such methods do not seem available for the curl–curl problem and a different approach is needed. Note further that there is no maximum principle for the curl–curl operator and, to our knowledge, no unique continuation principle applicable to our case. An approach different than for (1.1) is also required for the proof of $\Omega$-independence of $S_{\mathrm {curl}}$, see Section 5. Moreover concentration–compactness analysis for the curl operator is considerably different from that in [16, 21, 36]—see our approach in Section 3.

We would like to emphasize an important role of the analysis of nonlinear curl–curl problems from the physical point of view. Solutions u to nonlinear curl–curl equations describe the profiles of time-harmonic solutions $E(x,t)=u(x)\cos (\omega t)$ to the time-dependent nonlinear electromagnetic wave equation, which together with material constitutive laws and Maxwell equations, describes the exact propagation of electromagnetic waves in a nonlinear medium [1, 6, 31]. Since finding propagation exactly may be very difficult, there are several simplifications in the literature which rely on approximations of the nonlinear electromagnetic wave equation. The most prominent one is the scalar or vector nonlinear Schrödinger equation. For instance, one assumes that the term $\nabla (\mathop {\mathrm {div}}(u))$ in $\nabla \times (\nabla \times u)= \nabla (\mathop {\mathrm {div}}(u))-\Delta u$ is negligible and can be dropped, or one uses the so-called slowly varying envelope approximation. However, such simplifications may produce non-physical solutions; see [2, 11] and the references therein.

We also point out that the term $|u|^4u$ in (1.6) as well as in (1.7) below allows to consider the so-called quintic effect in nonlinear optics modelled by Maxwell equations. See for instance [1, 6, 14, 15, 23, 25, 31] and the references therein. We hope that our results will prompt further analytical studies of physical phenomena involving the quintic nonlinearity, for example the well-known cubic–quintic effect in nonlinear optics [14, 25].

Using our concentration–compactness result we are also able to treat the Brezis–Nirenberg problem [10] for the curl–curl operator

$$\begin{aligned} \nabla \times (\nabla \times u) + \lambda u = |u|^{4}u \qquad \text {in } \Omega , \end{aligned}$$

(1.7)

together with the so-called metallic boundary condition

$$\begin{aligned} \nu \times u = 0\qquad \text {on }\partial \Omega . \end{aligned}$$

(1.8)

Here $\nu :\partial \Omega \rightarrow \mathbb {R}^3$ is the exterior normal and $\Omega \subset \mathbb {R}^3$ is a bounded domain. This boundary condition is natural in the theory of Maxwell equations and it holds when $\Omega$ is surrounded by a perfect conductor. If the boundary of $\Omega$ is not of class ${{\mathcal {C}}}^{1}$, then we assume (1.8) is satisfied in a generalized sense by which we mean u is in the space $W^6_0(\mathrm {curl};\Omega )$ defined in (1.2). Weak solutions to (1.7)–(1.8) correspond to critical points of the associated energy functional $J_\lambda : W^6_0(\mathrm {curl};\Omega )\rightarrow \mathbb {R}$ given by

$$\begin{aligned} J_\lambda (u){:=}\frac{1}{2}\int _\Omega |\nabla \times u|^2\, \mathrm{d}x + \frac{\lambda }{2}\int _{\Omega } |u|^2\, \mathrm{d}x-\frac{1}{6}\int _\Omega |u|^6\, \mathrm{d}x. \end{aligned}$$

(1.9)

Recall from [7, 23] that the spectrum of the curl–curl operator in $H_0(\mathrm {curl};\Omega ) {:=} W^2_0(\mathrm {curl};\Omega )$ consists of the eigenvalue $\lambda _0=0$ with infinite multiplicity and of a sequence of eigenvalues

$$\begin{aligned} 0<\lambda _1\le \lambda _2\le \dots \le \lambda _k\rightarrow \infty \end{aligned}$$

with corresponding finite multiplicities $m(\lambda _k)\in \mathbb {N}$. Let ${{\mathcal {N}}}_\lambda$ be the generalized Nehari manifold for $J_\lambda$ (see (6.1) for the definition), and for $\lambda \le 0$ let

$$\begin{aligned} c_\lambda {:=} \inf _{{{\mathcal {N}}}_\lambda } J_\lambda . \end{aligned}$$

Denote the Lebesque measure of $\Omega$ by $|\Omega |$. We introduce the following condition:

• ($\Omega$) $\Omega$ is a bounded domain, either convex or with ${{\mathcal {C}}}^{1,1}$-boundary.

The reason for this assumption will be explained in the next section.

In domains $\Omega \ne \mathbb {R}^3$ we also introduce another constant, $\overline{S}_\mathrm {curl}(\Omega )$, such that the inequality

$$\begin{aligned} \int _{\Omega }|\nabla \times u|^2\, \mathrm{d}x\ge \overline{S}_\mathrm {curl}(\Omega )\inf _{w\in {{\mathcal {W}}}_{\Omega }}\Big (\int _{\Omega }|u+w|^6\,\mathrm{d}x\Big )^{\frac{1}{3}} \end{aligned}$$

(1.10)

holds for any $u\in W^6_0(\mathrm {curl};\Omega ){\setminus } \mathcal {W}_\Omega$, where ${{\mathcal {W}}}_\Omega {:=}\{w\in W^6_0(\mathrm {curl};\Omega ): \nabla \times w=0\}$, and $\overline{S}_\mathrm {curl}(\Omega )$ is largest with this property. As in (1.3), also here the above inequality trivially holds if $u\in {{\mathcal {W}}}_\Omega$. Although $\overline{S}_\mathrm {curl}(\Omega )$ seems to be more natural than $S_\mathrm {curl}(\Omega )$, we do not know whether it equals $S_\mathrm {curl}$. We are only able to prove the following result:

Theorem 1.3

Let $\Omega$ be a Lipschitz domain in $\mathbb {R}^3$, possibly unbounded, $\Omega \ne \mathbb {R}^3$. Then $S_\mathrm {curl}\ge \overline{S}_\mathrm {curl}(\Omega )$. If $\Omega$ satisfies $(\Omega )$, then $\overline{S}_\mathrm {curl}(\Omega )\ge S$.

Finally, the main result concerning the Brezis–Nirenberg problem for the curl–curl operator (1.7) reads as follows:

Theorem 1.4

Suppose $\Omega$ satisfies $(\Omega )$. Let $\lambda \in (-\lambda _\nu , -\lambda _{\nu -1}]$ for some $\nu \ge 1$. Then $c_\lambda >0$ and the following statements hold:

• (a) If $c_\lambda <c_0$, then there is a ground state solution to (1.7)–(1.8), that is, $c_\lambda$ is attained by a critical point of $J_\lambda$. A sufficient condition for this inequality to hold is $\lambda \in (-\lambda _\nu ,-\lambda _\nu + \overline{S}_\mathrm {curl}(\Omega )|\Omega |^{-2/3})$.

• (b) There exists $\varepsilon _\nu \ge \overline{S}_\mathrm {curl}(\Omega )|\Omega |^{-2/3}$ such that $c_\lambda$ is not attained for $\lambda \in (-\lambda _{\nu }+\varepsilon _\nu ,-\lambda _{\nu -1}]$, and $c_\lambda =c_0$ for $\lambda \in [-\lambda _{\nu }+\varepsilon _\nu ,-\lambda _{\nu -1}]$. We do not exclude that $\varepsilon _\nu >\lambda _\nu -\lambda _{\nu -1}$, so these intervals may be empty.

• (c) $c_\lambda \rightarrow 0$ as $\lambda \rightarrow -\lambda _\nu ^-$, and the function

  $$\begin{aligned} (-\lambda _{\nu },-\lambda _{\nu }+\varepsilon _\nu ]\cap (-\lambda _{\nu },-\lambda _{\nu -1}]\ni \lambda \mapsto c_\lambda \in (0,+\infty ) \end{aligned}$$

  is continuous and strictly increasing.

• (d) There are at least $\#\big \{k: -\lambda _k<\lambda <-\lambda _k +\frac{1}{3}\overline{S}_\mathrm {curl}(\Omega )|\Omega |^{-\frac{2}{3}}\big \}$ pairs of solutions $\pm u$ to (1.7)–(1.8).

Note that if $\lambda$ is as in (a), then the relation $-\lambda _k<\lambda <-\lambda _k +\frac{1}{3}\overline{S}_\mathrm {curl}(\Omega )|\Omega |^{-\frac{2}{3}}$ holds for $k=\nu ,\ldots ,\nu +m-1$ where m is the multiplicity of $\lambda _\nu$ but it may also hold for some k with $\lambda _k>\lambda _\nu$.

The above result is known for cylindrically symmetric domains where it is possible to reduce the curl–curl operator to a positive definite one, see [23]. However, the solution obtained there is a ground state in a subspace of functions having cylindric symmetry and we do not know whether it is a ground state in the full space.

Let us recall from earlier work that the main difficulties when treating J and $J_\lambda$, also in the subcritical case, are that these functionals are strongly indefinite, that is, they are unbounded from above and from below, even on subspaces of finite codimension. Moreover, the quadratic part of J has infinite-dimensional kernel and $J', J'_\lambda$ are not (sequentially) weak-to-weak$^*$ continuous, that is $u_n\rightharpoonup u$ does not imply that $J'_\lambda (u_n)\varphi \rightarrow J'_\lambda (u)\varphi$ for all $\varphi \in {{\mathcal {C}}}_0^{\infty }(\Omega ,\mathbb {R}^3)$. This lack of continuity is caused by the fact that $W^p_0(\mathrm {curl};\Omega )$ is not (locally) compactly embedded in any Lebesgue space and we do not know whether necessarily $u_n\rightarrow u$ almost everywhere in $\Omega$. A consequence of this is that for a Palais–Smale sequence $u_n\rightharpoonup u$ it is not clear whether u is a critical point. In the subcritical case one can overcome these difficulties since either a variant of the Palais–Smale condition is satisfied or some compactness can be recovered on a suitable topological manifold, see for example [6, 22, 24]. In the critical case however, there are additional difficulties. In Section 3 we introduce a general concentration–compactness analysis for this case. We show that the topological manifold

$$\begin{aligned} \Big \{u\in W_0^6(\mathrm {curl};\mathbb {R}^3): \mathop {\mathrm {div}}(|u|^4u)=0\Big \} \end{aligned}$$

is locally compactly embedded in $L^p(\mathbb {R}^3,\mathbb {R}^3)$ for $1\le p<6$ and that if a sequence $(u_n)$ is contained in this manifold and $u_n\rightharpoonup u$, then $u_n\rightarrow u$ almost everywhere after passing to a subsequence. This result will play a crucial role because it implies that if such $(u_n)$ is a Palais–Smale sequence, then u is a solution for our equation. If the condition $\mathop {\mathrm {div}}(|u|^4u)=0$ is violated, the embedding need not be locally compact.

The paper is organized as follows: In Section 2 we introduce the functional setting and some notation. Section 3 concerns the concentration–compactness analysis as we have already mentioned. In Section 4 we prove Theorem 1.2, and in Section 5 we prove Theorems 1.1 and 1.3. The proof of Theorem 1.4 is contained in Section 6 whereas in Section 7 we state some open problems.

2 Functional Setting and Preliminaries

Throughout the paper we assume that $\Omega$ is a Lipschitz domain in $\mathbb {R}^3$ and $2\le p\le 2^*=6$. The curl of u, $\nabla \times u$, should be understood in the distributional sense. We shall look for solutions to (1.6) and (1.7)–(1.8) in the space $W_0^6(\mathrm {curl};\mathbb {R}^3)$ and $W^6_0(\mathrm {curl};\Omega )$ respectively. We introduce the subspaces

$$\begin{aligned} {{\mathcal {V}}}_\Omega&{:=}&\left\{ v\in W^6_0(\mathrm {curl};\Omega ): \int _\Omega \langle v,\varphi \rangle \,\mathrm{d}x=0 \text { for every } \varphi \in {{\mathcal {C}}}^\infty _0(\Omega ,\mathbb {R}^3) \text { with } \nabla \times \varphi =0 \right\} ,\\ {{\mathcal {W}}}_\Omega&{:=}&\left\{ w\in W^6_0(\mathrm {curl};\Omega ): \int _\Omega \langle w,\nabla \times \varphi \rangle \,\mathrm{d}x = 0 \text { for all }\varphi \in {{\mathcal {C}}}^\infty _0(\Omega ,\mathbb {R}^3)\right\} \\= & {} \{w\in W^6_0(\mathrm {curl};\Omega ): \nabla \times w=0 \text { in the sense of distributions}\}. \end{aligned}$$

The second one has already been defined in Section 1. Here and below $\langle .\,,.\rangle$ denotes the inner product in $\mathbb {R}^3$. If $\Omega =\mathbb {R}^3$, we shall usually write ${{\mathcal {V}}}$ and ${{\mathcal {W}}}$ for ${{\mathcal {V}}}_{\mathbb {R}^3}$ and ${{\mathcal {W}}}_{\mathbb {R}^3}$.

In the sequel $\Omega$ is always a Lipschitz domain and C denotes a generic positive constant which may vary from one equation to another.

In the subsections that follow we consider two cases.

2.1 $\Omega =\mathbb {R}^3$

Lemma 2.1

$W^p(\mathrm {curl};\mathbb {R}^3) = W_0^p(\mathrm {curl};\mathbb {R}^3)$ for each $p\in [2,6]$.

Proof

We show ${{\mathcal {C}}}_0^\infty (\mathbb {R}^3,\mathbb {R}^3)$ is dense in $W^p(\mathrm {curl};\mathbb {R}^3)$. Let $\chi _R\in {{\mathcal {C}}}_0^\infty (\mathbb {R}^3)$ be such that $|\nabla \chi _R|\le 2/R$, $\chi _R=1$ for $|x|\le R$ and $\chi _R= 0$ for $|x|\ge 2R$. Take $u=(u_1,u_2,u_3)\in W^p(\mathrm {curl};\mathbb {R}^3)$. Then $\chi _Ru\rightarrow u$ in $L^p(\mathbb {R}^3,\mathbb {R}^3)$ as $R\rightarrow \infty$. We have

$$\begin{aligned} \partial _i(\chi _Ru_j)-\partial _j(\chi _Ru_i) = (\partial _i\chi _R)u_j - (\partial _j\chi _R)u_i + \chi _R(\partial _iu_j-\partial _ju_i), \ i\ne j. \qquad \end{aligned}$$

(2.1)

If $p=2$, it is clear that $(\partial _i\chi _R)u_j\rightarrow 0$ in $L^2(\mathbb {R}^3)$. If $2<p\le 6$, then

$$\begin{aligned} \int _{\mathbb {R}^3}(\partial _i\chi _R)^2u_j^2\,\mathrm{d}x \le \left( \int _{R\le |x|\le 2R}|\partial _i\chi _R|^q\,\mathrm{d}x\right) ^{2/q}\left( \int _{R\le |x|\le 2R}|u_j|^p\,\mathrm{d}x\right) ^{2/p} \end{aligned}$$

where $q=2p/(p-2)\ge 3$. Since

$$\begin{aligned} \int _{R\le |x|\le 2R}|\partial _i\chi _R|^q\,\mathrm{d}x \le CR^{3-q}<+\infty , \end{aligned}$$

also here $(\partial _i\chi _R)u_j\rightarrow 0$ in $L^2(\mathbb {R}^3)$. As $\partial _iu_j-\partial _ju_i\in L^2(\mathbb {R}^3)$, it follows that the left-hand side in (2.1) tends to $\partial _iu_j-\partial _ju_i$ in $L^2(\mathbb {R}^3)$ as $R\rightarrow \infty$. Hence $\chi _R u \rightarrow u$ in $W^p(\mathrm {curl};\mathbb {R}^3)$ and functions of compact support are dense in $W^p(\mathrm {curl};\mathbb {R}^3)$.

Suppose now $u\in W^p(\mathrm {curl};\mathbb {R}^3)$ has a compact support. Clearly, $j_\varepsilon *u\rightarrow u$ in $L^p(\mathbb {R}^3,\mathbb {R}^3)$ as $\varepsilon \rightarrow 0$ where $j_\varepsilon$ is the standard mollifier. Since

$$\begin{aligned} \partial _i(j_\varepsilon *u_j)-\partial _j(j_\varepsilon *u_i) = j_\varepsilon *(\partial _iu_j-\partial _ju_i) \end{aligned}$$

(2.2)

and $\partial _iu_j-\partial _ju_i\in L^2(\mathbb {R}^3)$, the right-hand side above tends to $\partial _iu_j-\partial _ju_i$ in $L^2(\mathbb {R}^3)$ as $\varepsilon \rightarrow 0$. This completes the proof. $\square$

As usual, let ${{\mathcal {D}}}^{1,2}(\mathbb {R}^3,\mathbb {R}^3)$ denote the completion of ${{\mathcal {C}}}^{\infty }_0(\mathbb {R}^3,\mathbb {R}^3)$ with respect to the norm $|\nabla \cdot |_2$. The following Helmholtz decomposition holds (see [22, 24]):

Lemma 2.2

$\mathcal {V}$ and ${{\mathcal {W}}}$ are closed subspaces of $W^6_0(\mathrm {curl};\mathbb {R}^3)$ and

$$\begin{aligned} W^6_0(\mathrm {curl};\mathbb {R}^3)=\mathcal {V}\oplus \mathcal {W}. \end{aligned}$$

(2.3)

Moreover, ${{\mathcal {V}}}\subset {{\mathcal {D}}}^{1,2}(\mathbb {R}^3,\mathbb {R}^3)$ and the norms $|\nabla \cdot |_2$ and $\Vert \cdot \Vert _{W^6(\mathrm {curl};\mathbb {R}^3)}$ are equivalent in $\mathcal {V}$.

We note that ${{\mathcal {W}}}$ is the closure of $\{\nabla \varphi : \varphi \in {{\mathcal {C}}}_0^\infty (\mathbb {R}^3)\}$. Indeed, if $w\in {{\mathcal {W}}}$, then $\nabla \times w = 0$, hence we can find $\varphi _n$ such that $\nabla \varphi _n\rightarrow w$ and $\nabla \varphi _n\in {{\mathcal {C}}}_0^\infty (\mathbb {R}^3,\mathbb {R}^3)$ [22, 24]. Since $\nabla \varphi _n=0$ outside of some ball, $\varphi _n$ is constant there and we may assume this constant is 0.

2.2 $\Omega$ Bounded

Recall $H_0(\mathrm {curl};\Omega ) {:=} W^2_0(\mathrm {curl};\Omega )$ and note that

$$\begin{aligned} \begin{aligned} {{\mathcal {V}}}_\Omega&\subset \left\{ u\in H_0(\mathrm {curl};\Omega ): \mathop {\mathrm {div}}(u)\in L^2(\Omega ,\mathbb {R}^3)\right\} . \end{aligned} \end{aligned}$$

Here we have used the fact that if $\varphi$ in the definition of ${{\mathcal {V}}}_\Omega$ is supported in a ball, then $\varphi =\nabla \psi$ for some $\psi$ and hence $u\in {{\mathcal {V}}}_\Omega$ implies $\mathop {\mathrm {div}}(u)=0$. It follows from [3, 12] that ${{\mathcal {V}}}_\Omega$ is continuously embedded in $H^{s}(\Omega ,\mathbb {R}^3)$ for some $s\in [1/2,1]$, hence compactly in $L^2(\Omega ,\mathbb {R}^3)$. If, in addition $\Omega$ satisfies $(\Omega )$, then ${{\mathcal {V}}}_\Omega$ is continuously embedded in $H^{1}(\Omega ,\mathbb {R}^3)$, hence compactly in $L^p(\Omega ,\mathbb {R}^3)$ for $1\le p<6$ and continuously in $L^6(\Omega ,\mathbb {R}^3)$. This implies, in particular, that

$$\begin{aligned} {{\mathcal {V}}}_\Omega =\left\{ v\in H_0(\mathrm {curl};\Omega ): \int _\Omega \langle v,\varphi \rangle \,\mathrm{d}x=0 \text { for every } \varphi \in {{\mathcal {C}}}^\infty _0(\Omega ,\mathbb {R}^3) \text { with } \nabla \times \varphi =0 \right\} \end{aligned}$$

(2.4)

is a Hilbert space with inner product

$$\begin{aligned} (v,z) = \int _{\Omega }\langle \nabla \times v,\nabla \times z\rangle \,\mathrm{d}x \equiv \int _{\Omega }\langle \nabla v,\nabla z\rangle \,\mathrm{d}x. \end{aligned}$$

Observe that the right-hand side of (2.4) is a closed linear subspace of $W^6_0(\mathrm {curl};\Omega )$ as a consequence of $(\Omega )$. Using this, it follows from the decomposition in [18, Theorem 4.21(c)] that also here there is a Helmholtz type decomposition

$$\begin{aligned} W^6_0(\mathrm {curl};\Omega ) = {{\mathcal {V}}}_\Omega \oplus {{\mathcal {W}}}_\Omega \end{aligned}$$

and that

$$\begin{aligned} \int _{\Omega }\langle v,w\rangle \,\mathrm{d}x = 0 \quad \text {if } v\in {{\mathcal {V}}}_\Omega ,\ w\in {{\mathcal {W}}}_\Omega \end{aligned}$$

which means that ${{\mathcal {V}}}_\Omega$ and ${{\mathcal {W}}}_\Omega$ are orthogonal in $L^2(\Omega ,\mathbb {R}^3)$. In $W_0^6(\mathrm {curl};\Omega )=\mathcal {V}_\Omega \oplus \mathcal {W}_\Omega$ we can use the norm

$$\begin{aligned} \Vert v+w\Vert {:=}\big ((v,v)+|w|_6^2\big )^{\frac{1}{2}}, \quad v\in {{\mathcal {V}}}_\Omega ,\ w\in {{\mathcal {W}}}_\Omega \end{aligned}$$

which is equivalent to $\Vert \cdot \Vert _{W_0^6(\mathrm {curl};\Omega )}$ if ($\Omega$) is satisfied.

According to [13, Theorem IX.2] or [26, Theorem 3.33], there is a continuous tangential trace operator $\gamma _t:H(\mathrm {curl};\Omega ) {:=} W^2(\mathrm {curl};\Omega ) \rightarrow H^{-1/2}(\partial \Omega )$ such that

$$\begin{aligned} \gamma _t(u)=\nu \times u|_{\partial \Omega }\qquad \text {for any } u\in {{\mathcal {C}}}^{\infty }({\overline{\Omega }},\mathbb {R}^3) \end{aligned}$$

and

$$\begin{aligned} H_0(\mathrm {curl};\Omega )=\{u\in H(\mathrm {curl};\Omega ): \gamma _t(u)=0\}. \end{aligned}$$

Hence any vector field $u\in W^6_0(\mathrm {curl};\Omega )=\mathcal {V}_\Omega \oplus \mathcal {W}_\Omega \subset H_0(\mathrm {curl};\Omega )$ satisfies the metallic boundary condition (1.8).

Denote the subspace of all gradient vector fields in $W_0^{1,6}(\Omega )$ by $\nabla W_0^{1,6}(\Omega )$. Clearly, $\nabla W_0^{1,6}(\Omega )\subset {{\mathcal {W}}}_\Omega$. However, for general domains the subspace $\{w\in \mathcal {W}_\Omega : \mathop {\mathrm {div}}(w)=0\}$ may be nontrivial and hence $\nabla W^{1,6}_0(\Omega )\subsetneq {{\mathcal {W}}}_\Omega$, see [7, pp. 4314 and 4315] and [26, Theorem 3.42].

Lemma 2.3

It holds that ${{\mathcal {W}}}_\Omega =W^6_0(\mathrm {curl};\Omega )\cap {{\mathcal {W}}}= W^6_0(\mathrm {curl};\Omega )\cap \nabla W^{1,6}(\Omega )$. If $\partial \Omega$ is connected, then ${{\mathcal {W}}}_\Omega =\nabla W^{1,6}_0(\Omega )$. If $\Omega$ is unbounded, ${{\mathcal {W}}}_\Omega =W^6_0(\mathrm {curl};\Omega )\cap {{\mathcal {W}}}$ still holds.

Proof

Let $w\in {{\mathcal {W}}}_{\Omega }$ and take a sequence $(\varphi _n)\subset {{\mathcal {C}}}_0^{\infty }(\Omega ,\mathbb {R}^3)$ such that $\varphi _n\rightarrow w$ in $W_0^6(\mathrm {curl};\Omega )$. Extend $\varphi _n$ by 0 in $\mathbb {R}^3{\setminus }\Omega$ and note that $(\varphi _n)$ is a Cauchy sequence, so $\varphi _n\rightarrow \widetilde{w}$ in $W^6_0(\mathbb {R}^3,\mathbb {R}^3)$ where $\widetilde{w}|_\Omega =w$ and $\widetilde{w}=0$ in $\mathbb {R}^3{\setminus }\Omega$. As

$$\begin{aligned}&\int _{\mathbb {R}^3}\langle \widetilde{w},\nabla \times \psi \rangle \,\mathrm{d}x=\lim _{n\rightarrow \infty } \int _{\mathbb {R}^3}\langle \varphi _n,\nabla \times \psi \rangle \,\mathrm{d}x\\&\quad = \lim _{n\rightarrow \infty } \int _{\mathbb {R}^3}\langle \nabla \times \varphi _n,\psi \rangle \,\mathrm{d}x\le \lim _{n\rightarrow \infty } |\nabla \times \varphi _n|_2|\psi |_2=0 \end{aligned}$$

for any $\psi \in {{\mathcal {C}}}_0^{\infty }(\mathbb {R}^3,\mathbb {R}^3)$, it follows that $\widetilde{w}\in {{\mathcal {W}}}$. Moreover, since $\widetilde{w}\in L^6(\mathbb {R}^3,\mathbb {R}^3)$ and $\nabla \times \widetilde{w} =0$, in view of [19, Lemma 1.1] we obtain $\widetilde{w}=\nabla \psi$ for some $\psi \in W^{1,6}_{loc}(\mathbb {R}^3)$. Therefore $w=\nabla \psi |_{\Omega }\in \nabla W^{1,6}(\Omega )$. Clearly, $W^6_0(\mathrm {curl};\Omega )\cap {{\mathcal {W}}}$ and $W^6_0(\mathrm {curl};\Omega )\cap \nabla W^{1,6}(\Omega )$ are contained in ${{\mathcal {W}}}_\Omega$.

Suppose that $\partial \Omega$ is connected. Similarly as above, we obtain $w=\nabla \psi$ for some $\psi \in W^{1,6}(\Omega )$ and the surface gradient

$$\begin{aligned} \nabla _S \psi =(\nu \times \nabla \psi )\times \nu =0. \end{aligned}$$

Therefore we may assume that $\psi \in W^{1,6}_0(\Omega )$, cf. [26, Theorem 4.3 and Remark 4.4]. $\square$

3 General Concentration–Compactness Analysis in $\mathbb {R}^N$

In this, self-contained, section we have $N\ge 3$ and we work in subspaces of $L^{2^*}(\mathbb {R}^N,\mathbb {R}^N)$ where $2^* {:=} 2N/(N-2)$.

Let $\Omega$ be a domain in $\mathbb {R}^N$, ${{\mathcal {V}}}$ a closed subspace of ${{\mathcal {D}}}^{1,2}(\mathbb {R}^N,\mathbb {R}^N)$ and

$$\begin{aligned} \mathcal {W}{:=}\big \{ w=(w_1,...,w_N)\in L^{2^*}(\Omega ,\mathbb {R}^N): \nabla \times w = 0\big \} \end{aligned}$$

(3.1)

where $\nabla \times w$ denotes the skew-symmetric, matrix-valued distribution having $\partial _k w_l-\partial _l w_k\in {{\mathcal {D}}}'(\Omega )$ as matrix elements. So for $N=3$, ${{\mathcal {W}}}$ corresponds to ${{\mathcal {W}}}_\Omega$ in Section 2 but ${{\mathcal {V}}}$ may be a more general subspace. Note that $\nabla \times$ is the usual curl operator if $N=3$. Let Z be a finite-dimensional subspace of $L^{2^*}(\Omega ,\mathbb {R}^N)$ such that $Z\cap {{\mathcal {W}}}= \{0\}$ and put

$$\begin{aligned} \widetilde{{{\mathcal {W}}}} {:=} {{\mathcal {W}}}\oplus Z. \end{aligned}$$

Assume

• (F1) $F:\Omega \times \mathbb {R}^N\rightarrow \mathbb {R}$ is differentiable with respect to the second variable $u\in \mathbb {R}^N$ for almost every $x\in \Omega$, $F(x,0)=0$ and $f=\partial _uF:\Omega \times \mathbb {R}^N\rightarrow \mathbb {R}^N$ is a Carathéodory function (that is, f is measurable in $x\in \Omega$ for all $u\in \mathbb {R}^N$ and continuous in $u\in \mathbb {R}^N$ for almost every $x\in \Omega$);

• (F2) F is uniformly strictly convex with respect to $u\in \mathbb {R}^N$, that is, for any compact set $A\subset (\mathbb {R}^N\times \mathbb {R}^N){\setminus }\{(u,u):\;u\in \mathbb {R}^N\}$

  $$\begin{aligned} \inf _{\genfrac{}{}{0.0pt}{}{x\in \Omega }{(u_1,u_2)\in A}} \left( \frac{1}{2}\big (F(x,u_1)+F(x,u_2)\big )-F\left( x,\frac{u_1+u_2}{2}\right) \right) > 0; \end{aligned}$$

• (F3) There are $c_1,c_2>0$ and $a\in L^{N/2}(\Omega )$, $a\ge 0$, such that

  $$\begin{aligned} c_1|u|^{2^*}\le F(x,u)\quad \hbox {and }|f(x,u)|\le a(x)|u|+c_2|u|^{2^*-1} \end{aligned}$$

  for every $u\in \mathbb {R}^N$ and almost every $x\in \Omega$.

In view of (F2) and (F3), for any $v\in {{\mathcal {V}}}$ we find a unique $\widetilde{w}_\Omega (v)\in \widetilde{{{\mathcal {W}}}}$ such that

$$\begin{aligned} \int _{\Omega }F(x,v+\widetilde{w}_\Omega (v))\,\mathrm{d}x \le \int _{\Omega }F(x,v+\widetilde{w})\,\mathrm{d}x \quad \text {for all } \widetilde{w}\in \widetilde{{{\mathcal {W}}}}. \end{aligned}$$

(3.2)

This implies that

$$\begin{aligned} \int _{\Omega }\langle f(x,v+\widetilde{w}),\zeta \rangle \,\mathrm{d}x =0 \quad \text {for all } \zeta \in \widetilde{{{\mathcal {W}}}} \text { if and only if } \widetilde{w}=\widetilde{w}_\Omega (v). \end{aligned}$$

(3.3)

Denote the space of finite measures in $\mathbb {R}^N$ by ${{\mathcal {M}}}(\mathbb {R}^N)$.

Theorem 3.1

Assume that (F1)–(F3) are satisfied. Suppose $(v_n)\subset {{\mathcal {V}}}$, $v_n\rightharpoonup v_0$ in ${{\mathcal {V}}}$, $v_n\rightarrow v_0$ almost everywhere in $\mathbb {R}^N$, $|\nabla v_n|^2\rightharpoonup \mu$ and $|v_n|^{2^*}\rightharpoonup \rho$ in ${{\mathcal {M}}}(\mathbb {R}^N)$. Then there exists an at most countable set $I\subset \mathbb {R}^N$ and nonnegative weights $\{\mu _x\}_{x\in I}$, $\{\rho _x\}_{x\in I}$ such that

$$\begin{aligned} \mu \ge |\nabla v_0|^2+\sum _{x\in I}\mu _x\delta _x, \quad \rho = |v_0|^{2^*} + \sum _{x\in I}\rho _x\delta _x, \end{aligned}$$

and passing to a subsequence, $\widetilde{w}_\Omega (v_n)\rightharpoonup \widetilde{w}_\Omega (v_0)$ in $\widetilde{{{\mathcal {W}}}}$, $\widetilde{w}_\Omega (v_n)\rightarrow \widetilde{w}_\Omega (v_0)$ almost everywhere in $\Omega$ and in $L^p_{loc}(\Omega )$ for any $1\le p<2^*$.

Remark 3.2

We shall use this theorem in Sections 4 and 6. In Section 4 we have $\Omega =\mathbb {R}^3$ and $Z=\{0\}$, so $\widetilde{w} = w$ and we will write w(v) for $w_{\mathbb {R}^3}(v)$. In Section 6, where we treat a Brezis–Nirenberg problem, $\Omega$ will be bounded and Z the subspace of ${{\mathcal {V}}}_{\Omega }$ on which the quadratic part of $J_\lambda$ (see (1.9)) is negative semidefinite.

Proof of Theorem 3.1

Step 1. Let $\varphi \in {{\mathcal {C}}}_0^{\infty }(\mathbb {R}^N)$. By the Sobolev inequality,

$$\begin{aligned} \Big (\int _{\mathbb {R}^N}|\varphi |^{2^*}|v_n-v_0|^{2^*}\,\mathrm{d}x\Big )^{1/2^*}\le & {} S^{-1/2}\Big (\int _{\mathbb {R}^N}|\nabla [\varphi (v_n-v_0)]|^{2}\,\mathrm{d}x\Big )^{1/2} \nonumber \\= & {} S^{-1/2}\Big (\int _{\mathbb {R}^N}|\varphi |^2|\nabla (v_n-v_0)|^{2}\,\mathrm{d}x\Big )^{1/2}+o(1). \nonumber \\ \end{aligned}$$

(3.4)

Passing to the limit and using the Brezis–Lieb lemma [9, 36] on the left-hand side above we obtain

$$\begin{aligned} \Big (\int _{\mathbb {R}^N}|\varphi |^{2^*}d{\bar{\rho }}\Big )^{1/2^*} \le S^{-1/2}\Big (\int _{\mathbb {R}^N}|\varphi |^2\,d\bar{\mu }\Big )^{1/2} \end{aligned}$$

(3.5)

where $\bar{\mu }{:=}\mu -|\nabla v_0|^2$ and $\bar{\rho }{:=}\rho -|v_0|^{2^*}$. Set $I=\{x\in \mathbb {R}^N: \mu _x{:=}\mu (\{x\})>0\}$. Since $\mu$ is finite and $\mu , {\bar{\mu }}$ have the same singular set, I is at most countable and $\mu \ge |\nabla v_0|^2+\sum _{x\in I}\mu _x\delta _x$. As in the proof of Theorem 1.9 in [16] it follows from (3.5) that ${\bar{\rho }} = \sum _{x\in I}\rho _x\delta _x$, see also Proposition 4.2 in [35]. So $\mu$ and $\rho$ are as claimed.

Step 2. Using (F3) and (3.2) we infer that

$$\begin{aligned} c_1|v_n+\widetilde{w}_\Omega (v_n)|_{2^*}^{2^*}\le & {} \int _{\Omega } F(x,v_n+\widetilde{w}_\Omega (v_n))\le \int _{\Omega } F(x,v_n)\,\mathrm{d}x \\\le & {} c_2 |v_n|_{2^*}^{2^*} + |a|_{N/2}|v_n|_{2^*}^2, \end{aligned}$$

and since the right-hand side above is bounded, so is $(|\widetilde{w}_\Omega (v_n)|_{2^*})$. Hence, up to a subsequence, $\widetilde{w}_\Omega (v_n)\rightharpoonup \widetilde{w}_0$ for some $\widetilde{w}_0$. Write $\widetilde{w}_\Omega (v_n) = w_n+z_n$, $\widetilde{w}_0=w_0+z_0$ where $w_n,w_0\in {{\mathcal {W}}}$ and $z_n,z_0\in Z$. We shall show that $\widetilde{w}_\Omega (v_n)\rightarrow \widetilde{w}_0$ almost everywhere in $\Omega$ after taking subsequences. Obviously, we may assume $z_n\rightarrow z_0$ in Z and almost everywhere in $\Omega$.

We can find a sequence of open balls $(B_{l})_{l=1}^\infty$ such that $\Omega =\bigcup _{l=1}^\infty B_l$. Fix $l\ge 1$. In view of [19, Lemma 1.1] there exists $\xi _n\in W^{1,2^*}(B_l)$ such that $w_n=\nabla \xi _n$ and we may assume without loss of generality that $\int _{B_l}\xi _n\,\mathrm{d}x = 0$. Then by the Poincaré inequality,

$$\begin{aligned} \Vert \xi _n\Vert _{W^{1,2^*}(B_l)} \le C|w_n|_{L^{2^*}(B_l,\mathbb {R}^N)} \le C|w_n|_{2^*} \end{aligned}$$

and passing to a subsequence, $\xi _n\rightharpoonup \xi$ for some $\xi \in W^{1,2^*}(B_l)$. So $\xi _n\rightarrow \xi$ in $L^{2^*}(B_l)$. Now take any $\varphi \in {{\mathcal {C}}}_0^{\infty }(B_l)$. Since $\nabla (|\varphi |^{2^*}(\xi _n-\xi ))\in {{\mathcal {W}}}$, in view of (3.3) we get

$$\begin{aligned} \int _{\Omega }\langle f(x,v_n+\widetilde{w}_\Omega (v_n)),\nabla (|\varphi |^{2^*} (\xi _n-\xi ))\rangle \,\mathrm{d}x = 0, \end{aligned}$$

that is,

$$\begin{aligned}&\int _{\Omega }|\varphi |^{2^*}\langle f(x,v_n+\widetilde{w}_\Omega (v_n)),w_n-\nabla \xi \rangle \,\mathrm{d}x\\&=\int _{\Omega }\langle f(x,v_n+\widetilde{w}_\Omega (v_n)),\nabla (|\varphi |^{2^*})(\xi -\xi _n)\rangle \,\mathrm{d}x, \end{aligned}$$

where the right-hand side tends to 0 as $n\rightarrow \infty$. Since $w_n\rightharpoonup \nabla \xi$ in $L^{2^*}(B_l)$,

$$\begin{aligned} \int _{\Omega }|\varphi |^{2^*}\langle f(x,v_0+\nabla \xi +z_0),\, w_n-\nabla \xi \rangle \,\mathrm{d}x =o(1), \end{aligned}$$

hence, recalling that $\widetilde{w}_\Omega (v_n)=w_n+z_n$ and $z_n\rightarrow z_0$, we obtain

$$\begin{aligned}&\int _{\Omega }|\varphi |^{2^*}\langle f(x,v_n+\widetilde{w}_\Omega (v_n))-f(x,v_0+\nabla \xi +z_0),\, \widetilde{w}_\Omega (v_n)\nonumber \\&-\nabla \xi -z_0\rangle \,\mathrm{d}x =o(1). \end{aligned}$$

(3.6)

The convexity of F in u implies that

$$\begin{aligned} F\Big (x,\frac{u_1+u_2}{2}\Big )\ge F(x,u_1)+ \Big \langle f(x,u_1),\,\frac{u_2-u_1}{2}\Big \rangle \end{aligned}$$

and

$$\begin{aligned} F\Big (x,\frac{u_1+u_2}{2}\Big )\ge F(x,u_2)+\Big \langle f(x,u_2),\, \frac{u_1-u_2}{2}\Big \rangle . \end{aligned}$$

Adding these inequalities and using (F2), we obtain for any $k\ge 1$ and $|u_1-u_2|\ge \frac{1}{k}$, $|u_1|, |u_2| \le k$ that

$$\begin{aligned}&m_{k} \le \frac{1}{2}(F(x,u_1)+F(x,u_2)) - F\Big (x,\frac{u_1+u_2}{2}\Big )\nonumber \\&\le \frac{1}{4}\langle f(x,u_1)-f(x,u_2),\, u_1-u_2\rangle \end{aligned}$$

(3.7)

where

$$\begin{aligned} m_{k}{:=}\inf _{\begin{array}{c} x\in \Omega ,u_1,u_2\in \mathbb {R}^N\\ \frac{1}{k}\le |u_1-u_2|,\\ |u_1|,|u_2|\le k \end{array} }\; \frac{1}{2}(F(x,u_1)+F(x,u_2))-F\Big (x,\frac{u_1+u_2}{2}\Big )>0. \end{aligned}$$

(3.8)

Let

$$\begin{aligned} \Omega _{n,k}:= & {} \Big \{x\in \Omega : |v_n+\widetilde{w}_\Omega (v_n)-v_0-\nabla \xi -z_0|\ge \frac{1}{k} \\&\quad \hbox { and }|v_n+\widetilde{w}_\Omega (v_n)|,|v_0+\nabla \xi +z_0|\le k\Big \}. \end{aligned}$$

Taking into account (3.6) and using (F3), (3.7) and Hölder’s inequality, we get

$$\begin{aligned}&4m_k\int _{\Omega _{n,k}}|\varphi |^{2^*}\,\mathrm{d}x \\&\quad \le \int _{\Omega }|\varphi |^{2^*}\langle f(x,v_n+\widetilde{w}_\Omega (v_n))\\&\quad -f(x,v_0+\nabla \xi +z_0), \, v_n+\widetilde{w}_\Omega (v_n)-v_0-\nabla \xi -z_0\rangle \,\mathrm{d}x\\&\quad \le \int _{\Omega }|\varphi |^{2^*}\langle f(x,v_n+\widetilde{w}_\Omega (v_n))-f(x,v_0+\nabla \xi +z_0), \, v_n-v_0\rangle \,\mathrm{d}x +o(1)\\&\quad \le C \Big (\int _{\Omega }|\varphi |^{2^*}|v_n-v_0|^{2^*}\,\mathrm{d}x\Big )^{1/2^*} +o(1) = C \Big (\int _{\Omega }|\varphi |^{2^*}\,d\bar{\rho }\Big )^{1/2^*}+o(1), \end{aligned}$$

where k is fixed. Here we have used the fact that $\int _{\Omega }a(x)|v_n-v_0|^2\,\mathrm{d}x \rightarrow 0$ if $v_n\rightharpoonup v_0$ in $L^{2^*}(\Omega ,\mathbb {R}^N)$. Since $\varphi \in {{\mathcal {C}}}_0^{\infty }(B_l)$ is arbitrary,

$$\begin{aligned} 4m_k|\Omega _{n,k}\cap E|\le \big (\bar{\rho }(E)\big )^{1/2^*}+o(1) \end{aligned}$$

(3.9)

for any Borel set $E\subset B_l$. We find an open set $E_k\supset I$ such that $|E_k|<1/2^{k+1}$. Then, taking $E=B_l{\setminus } E_k$ in (3.9), we have $4m_k|\Omega _{n,k}\cap (B_l{\setminus } E_k)|=o(1)$ as $n\rightarrow \infty$ because $\mathrm {supp}({\bar{\rho }})\subset I$; hence we can find a sufficiently large $n_k$ such that $|\Omega _{n_k,k}\cap B_l|<1/2^k$ and we obtain

$$\begin{aligned} \Big |\bigcap _{j=1}^{\infty }\bigcup _{k=j}^{\infty }\Omega _{n_k,k}\cap B_l\Big | \le \lim _{j\rightarrow \infty }\sum _{k=j}^{\infty }|\Omega _{n_k,k}\cap B_l|\le \lim _{j\rightarrow \infty }\frac{1}{2^{j-1}}=0. \end{aligned}$$

If $x\notin \bigcap _{j=1}^{\infty }\bigcup _{k=j}^{\infty }\Omega _{n_k,k}$ and $x\in B_l$, then

$$\begin{aligned}&|v_{n_k}(x)+\widetilde{w}_\Omega (v_{n_k})(x)-v_0(x)-\nabla \xi (x)-z_0(x)|< \frac{1}{k},\\&\text{ or } |v_{n_k}(x)+\widetilde{w}_\Omega (v_{n_k})(x)|>k,\\&\text{ or } |v_0(x)+\nabla \xi (x)+z_0(x)|> k \end{aligned}$$

for all sufficiently large k. Since $v_{n_k}+\widetilde{w}_\Omega (v_{n_k})$ is bounded in $L^{2^*}(\Omega ,\mathbb {R}^N)$, the second and the third inequality above cannot hold on a set of positive measure for all large k. We infer that $v_{n_k}+\widetilde{w}_\Omega (v_{n_k})\rightarrow v_0+\nabla \xi +z_0$, hence $\widetilde{w}_\Omega (v_{n_k})\rightarrow \nabla \xi +z_0$ almost everywhere in $B_l$. Since $\widetilde{w}_\Omega (v_n)\rightharpoonup \widetilde{w}_0$, $\widetilde{w}_0=\nabla \xi +z_0$ almost everywhere in $B_l$. Now employing the diagonal procedure, we find a subsequence of $\widetilde{w}_\Omega (v_n)$ which converges to $\widetilde{w}_0$ almost everywhere in $\Omega =\bigcup _{l=1}^{\infty }B_l$.

Let $p\in [1,2^*)$. For $\Omega '\subset \Omega$ such that $|\Omega '|<+\infty$ we have

$$\begin{aligned}&\int _{\Omega '} |v_n-v_0\!+\!\widetilde{w}_\Omega (v_n)-\widetilde{w}_0|^p\,\mathrm {d}x \!\le \! |\Omega '|^{1-\frac{p}{2^*}}\Big (\int _{\Omega }|v_n-v_0+\widetilde{w}_\Omega (v_n)-\widetilde{w}_0|^{2^*}\,\mathrm {d}x\Big )^{\frac{p}{2^*}}, \end{aligned}$$

hence by the Vitali convergence theorem, $v_n-v_0+\widetilde{w}_\Omega (v_n)-\widetilde{w}_0\rightarrow 0$ in $L^p_{loc}(\Omega )$ after passing to a subsequence.

Step 3. We show that $\widetilde{w}_\Omega (v_0)=\widetilde{w}_0$. Take any $\widetilde{w}\in \widetilde{{{\mathcal {W}}}}$ and observe that, by the Vitali convergence theorem,

$$\begin{aligned} 0 = \int _{\Omega }\langle f(x,v_n+\widetilde{w}_\Omega (v_n)), \widetilde{w}\rangle \,\mathrm{d}x\rightarrow \int _{\Omega }\langle f(x,v_0+\widetilde{w}_0), \widetilde{w}\rangle \,\mathrm{d}x \end{aligned}$$

up to a subsequence. Now (3.3) implies that $\widetilde{w}_0=\widetilde{w}_\Omega (v_0)$ which completes the proof. $\square$

4 Problem in $\Omega =\mathbb {R}^3$ and Proof of Theorem 1.2

Let S be the best Sobolev constant for the embedding of ${{\mathcal {D}}}^{1,2}(\mathbb {R}^3)$ into $L^6(\mathbb {R}^3)$, see (1.1). It is clear that a minimizer w(u) in (3.2) exists uniquely for any $u\in W_0^6(\mathrm {curl};\Omega )$, not only for $u\in {{\mathcal {V}}}$. Here we have $F(x,u)=\frac{1}{6}|u|^6$ and $Z=\{0\}$. So by Lemma 2.2, $u+w(u)=v+w(v)\in {{\mathcal {V}}}\oplus {{\mathcal {W}}}$ for some $v\in {{\mathcal {V}}}$ and therefore

$$\begin{aligned} \inf _{w\in {{\mathcal {W}}}} \int _{\mathbb {R}^3} |u+w|^6\,\mathrm{d}x = \int _{\mathbb {R}^3}|u+w(u)|^6\,\mathrm{d}x = \int _{\mathbb {R}^3}|v+w(v)|^6\,\mathrm{d}x. \end{aligned}$$

(4.1)

Since $\mathop {\mathrm {div}}(v)=0$,

$$\begin{aligned} S_\mathrm {curl}= \inf _{\begin{array}{c} u\in W_0^6(\mathrm {curl};\mathbb {R}^3) \\ \nabla \times u\ne 0 \end{array}} \frac{|\nabla \times u|_2^2}{|u+w(u)|_6^2} = \inf _{v\in {{\mathcal {V}}}{\setminus }\{0\}} \frac{|\nabla v|_2^2}{|v+w(v)|_6^2}. \end{aligned}$$

(4.2)

Lemma 4.1

$S_{\mathrm {curl}}\ge S$.

Proof

Given $\varepsilon >0$, by (4.2) we can find $v\ne 0$ such that

$$\begin{aligned} \int _{\mathbb {R}^3}|\nabla v|^2\, \mathrm{d}x\le (S_{\mathrm {curl}}+\varepsilon )\Big ( \int _{\mathbb {R}^3}|v+w(v)|^6\,\mathrm{d}x\Big )^{\frac{1}{3}}. \end{aligned}$$

(4.3)

Let $v=(v_1,v_2,v_3)$. By the Hölder inequality,

$$\begin{aligned} \int _{\mathbb {R}^3} v_1^2v_2^2v_3^2\,\mathrm{d}x \le \Big (\int _{\mathbb {R}^3}v_1^6\,\mathrm{d}x \int _{\mathbb {R}^3}v_2^6\,\mathrm{d}x \int _{\mathbb {R}^3}v_3^6\,\mathrm{d}x\Big )^{\frac{1}{3}} \end{aligned}$$

(4.4)

and

$$\begin{aligned} \int _{\mathbb {R}^3} v_i^4v_j^2\,\mathrm{d}x \le \Big (\int _{\mathbb {R}^3} v_i^6\Big )^{\frac{2}{3}}\Big (\int _{\mathbb {R}^3} v_j^6\,\mathrm{d}x\Big )^{\frac{1}{3}}, \quad i\ne j. \end{aligned}$$

(4.5)

Using this and the Sobolev inequality gives

$$\begin{aligned} \int _{\mathbb {R}^3}|\nabla v|^2\,\mathrm{d}x\ge S\sum _{i=1}^3\Big (\int _{\mathbb {R}^3}|v_i|^6\,\mathrm{d}x\Big )^{1/3}\ge S\Big (\int _{\mathbb {R}^3}|v|^6\,\mathrm{d}x\Big )^{1/3}, \end{aligned}$$

(4.6)

and since w(v) is a minimizer, we obtain using (4.3) and (4.6)

$$\begin{aligned} \int _{\mathbb {R}^3}|\nabla v|^2\, \mathrm{d}x\le & {} (S_{\mathrm {curl}}+\varepsilon ) \Big (\int _{\mathbb {R}^3}|v+w(v)|^6\,\mathrm{d}x\Big )^{\frac{1}{3}} \le (S_{\mathrm {curl}}+\varepsilon ) \Big (\int _{\mathbb {R}^3}|v|^6\,\mathrm{d}x\Big )^{\frac{1}{3}} \nonumber \\\le & {} (S_\mathrm {curl}+\varepsilon )/S\int _{\mathbb {R}^3}|\nabla v|^2\, \mathrm{d}x. \end{aligned}$$

(4.7)

Hence $S_{\mathrm {curl}}+\varepsilon \ge S$ for all $\varepsilon >0$ and the conclusion follows. $\square$

Next we look for ground states for the curl–curl problem (1.6), that is, nontrivial solutions with least possible associated energy J given by (1.4). Throughout the rest of the paper we shall make repeated use of the following fact:

Lemma 4.2

Let $\lambda >0$. Then $w(\lambda u) = \lambda w(u)$. Similarly, if $\Omega$ is a proper subset of $\mathbb {R}^3$, then $w_\Omega (\lambda u) = \lambda w_\Omega (u)$.

Proof

We prove this for $w_\Omega$. Using the minimizing property of $w_\Omega (u)$ we obtain

$$\begin{aligned} \lambda ^6\int _{\Omega }|u+w_\Omega (u)|^6\,\mathrm{d}x= & {} \int _{\Omega }|\lambda u+\lambda w_\Omega (u)|^6\,\mathrm{d}x \ge \int _{\Omega }|\lambda u+w_\Omega (\lambda u)|^6\,\mathrm{d}x \\= & {} \lambda ^6\int _{\Omega }|u+w_\Omega (\lambda u)/\lambda |^6\,\mathrm{d}x \ge \lambda ^6\int _{\Omega }|u+w_\Omega (u)|^6\,\mathrm{d}x. \end{aligned}$$

Since the minimizer is unique, $w_\Omega (u) = w_\Omega (\lambda u)/\lambda$ as claimed. $\square$

Lemma 4.3

Let ${{\mathcal {N}}}$ be the set defined in (1.5). Then

$$\begin{aligned} {{\mathcal {N}}}=\{u\in W^6_0(\mathrm {curl};\mathbb {R}^3){\setminus }{{\mathcal {W}}}: J'(u)u=0 \text { and } J'(u)|_{{\mathcal {W}}}=0\}. \end{aligned}$$

(4.8)

Proof

The first condition in (1.5) is equivalent to $J'(u)u=0$. The second condition is satisfied because $\mathop {\mathrm {div}}(|u|^4u)=0$ if and only if $\int _{\mathbb {R}^3}\langle |u|^4u,\nabla \varphi \rangle \,\mathrm{d}x = 0$ for all $\varphi \in {{\mathcal {C}}}_0^\infty (\mathbb {R}^3)$ and each element of ${{\mathcal {W}}}$ can be approximated by such $\varphi$, see the comment preceding Section 2.2. $\square$

By Lemma 2.2, $W_0^6(\mathrm {curl};\mathbb {R}^3) = {{\mathcal {V}}}\oplus {{\mathcal {W}}}$. It follows from (3.2) and (3.3) that if $v\in {{\mathcal {V}}}$, then $J'(v+w(v))|_{{\mathcal {W}}}=0$, and as

$$\begin{aligned} J(t(v+w(v))) = \frac{t^2}{2}\int _{\mathbb {R}^3}|\nabla v|^2\,\mathrm{d}x - \frac{t^6}{6}\int _{\mathbb {R}^3}|v+w(v)|^6\,\mathrm{d}x, \end{aligned}$$

(4.9)

there is a unique $t(v)>0$ such that

$$\begin{aligned} m(v){:=}t(v)(v+w(v))\in {{\mathcal {N}}}\quad \hbox {for }v\in {{\mathcal {V}}}{\setminus }\{0\}. \end{aligned}$$

(4.10)

We note that

$$\begin{aligned} J(m(v)) \ge J(t(v+w)) \quad \text {for all } t>0 \text { and }w\in {{\mathcal {W}}}. \end{aligned}$$

(4.11)

Since $J(m(v)) \ge J(v)$ and there exist $a,r>0$ such that $J(v)\ge a$ if $\Vert v\Vert =r$, ${{\mathcal {N}}}$ is bounded away from ${{\mathcal {W}}}$ and hence closed.

Lemma 4.4

The mapping $m: {{\mathcal {V}}}{\setminus }\{0\}\rightarrow {{\mathcal {N}}}$ given by (4.10) is continuous.

Proof

Let $v_n\rightarrow v_0\ne 0$ in ${{\mathcal {V}}}$. Since

$$\begin{aligned} \int _{\mathbb {R}^3}|v_n+w(v_n)|^6\,\mathrm{d}x \le \int _{\mathbb {R}^3}|v_n|^6\,\mathrm{d}x, \end{aligned}$$

(4.12)

it follows that $(w(v_n))$ is bounded and it is then clear from (4.9) that so is $(t(v_n))$. Hence we may assume $t(v_n)\rightarrow t_0$ and $w(v_n)\rightharpoonup w_0$ in $L^6(\mathbb {R}^3,\mathbb {R}^3)$. By the weak sequential lower semicontinuity of the second integral in (4.9) and by (4.11),

$$\begin{aligned}&J(t_0(v_0+w_0))\ge \limsup _{n\rightarrow \infty } J(t(v_n)(v_n+w(v_n)))\\&\ge \limsup _{n\rightarrow \infty } J(t_0(v_n+w_0)) =J(t_0(v_0+w_0)). \end{aligned}$$

So $w(v_n)\rightarrow w_0$ and since ${{\mathcal {N}}}$ is closed, $t_0(v_0+w_0) = t(v_0)(v_0+w(v_0)) = m(v_0)$. $\square$

Now it is easily seen that $m|_{{\mathcal {S}}}:{{\mathcal {S}}}{:=}\{v\in {{\mathcal {V}}}: \Vert v\Vert =1\}\rightarrow {{\mathcal {N}}}$ is a homeomorphism with the inverse $u=v+w(v)\mapsto v/\Vert v\Vert$. Note that ${{\mathcal {N}}}$ is an infinite-dimensional topological manifold of infinite codimension. Although J is of class ${{\mathcal {C}}}^2$, we do not know whether ${{\mathcal {N}}}$ is of class ${{\mathcal {C}}}^1$. However, repeating the argument in [22, Proposition 4.4(b)] or [32, Proposition 2.9] we see that $J\circ m|_{{{\mathcal {S}}}}:{{\mathcal {S}}}\rightarrow \mathbb {R}$ is of class ${{\mathcal {C}}}^1$ and is bounded from below by the constant $a>0$ introduced above. By the Ekeland variational principle [36, Theorem 8.5], there is a Palais–Smale sequence $(v_n)\subset {{\mathcal {S}}}$ such that

$$\begin{aligned} (J\circ m)(v_n)\rightarrow \inf _{{{\mathcal {S}}}} J\circ m=\inf _{{{\mathcal {N}}}}J\ge a>0. \end{aligned}$$

(4.13)

It follows from [22, Proposition 4.4(b)] again or from [32, Corollary 2.10] that $(m(v_n))$ is a Palais–Smale sequence for J on ${{\mathcal {N}}}$, so in particular, $J'(m(v_n))\rightarrow 0$ as $n\rightarrow \infty$. See also an abstract critical point theory on the generalized Nehari manifold in [6, Section 4] and in [7, Section 4].

For $s>0$, $y\in \mathbb {R}^3$ and $u:\mathbb {R}^3\rightarrow \mathbb {R}^3$ we denote $T_{s,y}(u){:=} s^{1/2}u(s\cdot +y))$. The next lemma is a special case of [29, Theorem 1], see also [34, Lemma 5.3].

Lemma 4.5

Suppose that $(v_n)\subset {{\mathcal {D}}}^{1,2}(\mathbb {R}^3,\mathbb {R}^3)$ is bounded. Then $v_n\rightarrow 0$ in $L^6(\mathbb {R}^3,\mathbb {R}^3)$ if and only if $T_{s_n,y_n}(v_n)\rightharpoonup 0$ in ${{\mathcal {D}}}^{1,2}(\mathbb {R}^3,\mathbb {R}^3)$ for all $(s_n)\subset \mathbb {R}^+$ and $(y_n)\subset \mathbb {R}^3$.

Observe that the above lemma in [29] is expressed in terms of the space $H^{1,2}$. However, in the notation of [29], this is the same space as our ${{\mathcal {D}}}^{1,2}$.

Lemma 4.6

$T_{s,y}$ is an isometric isomorphism of $W_0^6(\mathrm {curl};\mathbb {R}^3)$ which leaves the functional J and the subspaces ${{\mathcal {V}}}, {{\mathcal {W}}}$ invariant. In particular, $w(T_{s,y}u)=T_{s,y}w(u)$.

The proof is by an explicit (and simple) computation.

Lemma 4.7

Suppose $u+w(u)\in {{\mathcal {N}}}$. Then

$$\begin{aligned} \frac{|\nabla \times u|_2^2}{|u+w(u)|_6^2} = A \quad \text {if and only if} \quad J(u+w(u)) = \frac{1}{3} A^{3/2}. \end{aligned}$$

In particular, $\inf _{{\mathcal {N}}}J = \frac{1}{3}S_\mathrm {curl}^{3/2}$.

Proof

Since $u+w(u)\in {{\mathcal {N}}}$, $J'(u)u=0$, that is $|\nabla \times u|_2^2 = |u+w(u)|_6^6$. Hence

$$\begin{aligned} \frac{|\nabla \times u|_2^2}{|u+w(u)|_6^2} = |u+w(u)|_6^4 \quad \text {and} \quad J(u+w(u)) = \frac{1}{3}|u+w(u)|_6^6. \end{aligned}$$

$\square$

Proof of Theorem 1.2

We prove part (b) first. Take a minimizing sequence $(u_n) = (m(v_n))\subset {{\mathcal {N}}}$ constructed above and write $u_n=t(v_n)(v_n+w(v_n)) = v_n'+w(v_n')\in {{\mathcal {V}}}\oplus {{\mathcal {W}}}$. As

$$\begin{aligned} J(u_n) = J(u_n) -\frac{1}{6}J'(u_n)u_n = \frac{1}{3}|\nabla \times u_n|^2_2 = \frac{1}{3}|\nabla v_n'|^2_2 \end{aligned}$$

(4.14)

and $|\nabla \cdot |_2$ is an equivalent norm in ${{\mathcal {V}}}$, $(v_n')$ is bounded. We also have

$$\begin{aligned} J(u_n) = J(u_n) -\frac{1}{2}J'(u_n)u_n = \frac{1}{3}|u_n|^6_6. \end{aligned}$$

(4.15)

Since $J(u_n)$ is bounded away from 0, $|u_n|_6\not \rightarrow 0$ and hence by (4.12), $|v_n'|_6\not \rightarrow 0$. Therefore, passing to a subsequence and using Lemma 4.5, $\widetilde{v}_n{:=}T_{s_n,y_n}(v_n')\rightharpoonup v_0$ for some $v_0\ne 0$, $(s_n)\subset \mathbb {R}^+$ and $(y_n)\subset \mathbb {R}^3$. Taking subsequences again we also have that $\widetilde{v}_n\rightarrow v_0$ almost everywhere in $\mathbb {R}^3$ and in view of Theorem 3.1, $w(\widetilde{v}_n)\rightharpoonup w(v_0)$ and $w(\widetilde{v}_n)\rightarrow w(v_0)$ almost everywhere in $\mathbb {R}^3$. We set $u{:=}v_0+w(v_0)$ and by Lemma 4.6 we may assume without loss of generality that $s_n=1$ and $y_n=0$. So if $z\in W_0^6(\mathrm {curl};\mathbb {R}^3)$, then using weak and almost everywhere convergence,

$$\begin{aligned} J'(u_n)z = \int _{\mathbb {R}^3}\langle \nabla \times u_n, \nabla \times z\rangle \,\mathrm{d}x -\int _{\mathbb {R}^3}\langle |u_n|^4u_n,z\rangle \,\mathrm{d}x \rightarrow J'(u)z. \end{aligned}$$

Here we have used that $|u_n|^4u_n\rightharpoonup \zeta$ in $L^{6/5}(\mathbb {R}^3,\mathbb {R}^3)$ for some $\zeta$ but since $|u_n|^4u_n\rightarrow |u|^4u$ almost everywhere, $\zeta = |u|^4u$. Thus u is a solution to (1.6). To show it is a ground state, we note that using Fatou’s lemma,

$$\begin{aligned} \inf _{{\mathcal {N}}}J= & {} J(u_n)+o(1) = J(u_n)-\frac{1}{2}J'(u_n)u_n+o(1) = \frac{1}{3}|u_n|^6_6 + o(1) \\\ge & {} \frac{1}{3}|u|^6_6 + o(1) = J(u)-\frac{1}{2}J'(u)u+o(1) = J(u)+o(1). \end{aligned}$$

Hence $J(u)\le \inf _{{\mathcal {N}}}J$ and as a solution, $u\in {{\mathcal {N}}}$. It follows using Lemma 4.7 that $J(u) = \inf _{{\mathcal {N}}}J = \frac{1}{3}S_\mathrm {curl}^{3/2}$.

If u satisfies equality in (1.3), then $t(u)(u+w(u))\in {{\mathcal {N}}}$ and is a minimizer for $J|_{{\mathcal {N}}}$. But then the corresponding point v in ${{\mathcal {S}}}$ is a minimizer for $J\circ m|_{{\mathcal {S}}}$, see (4.13). So v is a critical point of $J\circ m|_{{\mathcal {S}}}$ and $m(v)=u$ is a critical point of J. This completes the proof of (b).

(a) By Lemma 4.1, $S_\mathrm {curl}\ge S$ and by part (b), there exists $u=v+w(v)$ for which $S_\mathrm {curl}$ is attained. Suppose $S_\mathrm {curl}=S$. Then all inequalities become equalities in (4.7) with $\varepsilon =0$, and therefore also in (4.6), but then $\int _{\mathbb {R}^3}|\nabla v_i|^2\,\mathrm{d}x = S|v_i|_6^2$ for $i=1,2,3$ and hence all $v_i$ are instantons, up to multiplicative constants. Since $v\ne 0$ and $\mathop {\mathrm {div}}(v)=0$, this is impossible. It follows that $S_\mathrm {curl}>S$. $\square$

5 Proof of Theorems 1.1 and 1.3

Let $\Omega$ be a Lipschitz domain in $\mathbb {R}^3$. Recall from Section 2 that we have the Helmholtz decompositions

$$\begin{aligned} W_0^6(\mathrm {curl};\mathbb {R}^3) = {{\mathcal {V}}}\oplus {{\mathcal {W}}}\quad \text {and}\quad W_0^6(\mathrm {curl};\Omega ) = {{\mathcal {V}}}_\Omega \oplus {{\mathcal {W}}}_\Omega \end{aligned}$$

(5.1)

where the second one holds if condition $(\Omega )$ in the introduction is satisfied. For $u\in W_0^6(\mathrm {curl};\Omega )$, denote the minimizer of

$$\begin{aligned} \int _{\Omega }|u+w|^6\,\mathrm{d}x, \quad w\in {{\mathcal {W}}}_\Omega \end{aligned}$$

by $w_\Omega (u)$ (cf. (4.1)) and, according to our notational convention, write w(u) for $w_{\mathbb {R}^3}(u)$. Recall from (1.3) the definition of $S_\mathrm {curl}(\Omega )$:

$$\begin{aligned} \int _{\mathbb {R}^3}|\nabla \times u|^2\,\mathrm{d}x \ge S_\mathrm {curl}(\Omega )\inf _{w\in {{\mathcal {W}}}}\left( \int _{\mathbb {R}^3}|u+w|^6\,\mathrm{d}x\right) ^{1/3}, \end{aligned}$$

where $u\in W_0^6(\mathrm {curl};\Omega ){\setminus }{{\mathcal {W}}}$ and $S_\mathrm {curl}(\Omega )$ is the largest constant with this property. By (5.1) we have $u=v+w\in {{\mathcal {V}}}\oplus {{\mathcal {W}}}$. We emphasize that although $u=0$ in $\mathbb {R}^3{\setminus }\overline{\Omega }$, v and w need not be 0 there. Note that $S_\mathrm {curl}({\Omega )}$ can be characterized as

$$\begin{aligned} S_\mathrm {curl}(\Omega ) = \inf _{\begin{array}{c} u\in W_0^6(\mathrm {curl};\Omega )\\ \nabla \times u\ne 0 \end{array}} \sup _{w\in {{\mathcal {W}}}} \frac{|\nabla \times u|_2^2}{|u+w|_6^2} = \inf _{\begin{array}{c} u\in W_0^6(\mathrm {curl};\Omega )\\ \nabla \times u\ne 0 \end{array}} \frac{|\nabla \times u|_2^2}{|u+w(u)|_6^2} \end{aligned}$$

(5.2)

(cf. (4.2)). In domains $\Omega \ne \mathbb {R}^3$ there is also another constant, $\overline{S}_\mathrm {curl}(\Omega )$, introduced in (1.10). Similarly as in (5.2), it can be characterized as

$$\begin{aligned} \overline{S}_\mathrm {curl}(\Omega ) = \inf _{\begin{array}{c} u\in W_0^6(\mathrm {curl};\Omega )\\ \nabla \times u\ne 0 \end{array}} \sup _{w\in {{\mathcal {W}}}_\Omega } \frac{|\nabla \times u|_2^2}{|u+w|_6^2} = \inf _{\begin{array}{c} u\in W_0^6(\mathrm {curl};\Omega )\\ \nabla \times u\ne 0 \end{array}} \frac{|\nabla \times u|_2^2}{|u+w_\Omega (u)|_6^2}. \end{aligned}$$

(5.3)

As we have noticed in the introduction, although this constant seems more natural, we do not know whether it equals $S_\mathrm {curl}$.

Lemma 5.1

The mapping $u\mapsto w_\Omega (u): L^6(\Omega ,\mathbb {R}^3)\rightarrow L^6(\Omega ,\mathbb {R}^3)$ is continuous ( $\Omega =\mathbb {R}^3$ is admitted).

Proof

Let $u_n\rightarrow u_0$. Since $(w_\Omega (u_n))$ is bounded, $w_\Omega (u_n)\rightharpoonup w_0$ after passing to a subsequence. By the maximality and uniqueness of $w_\Omega (\cdot )$,

$$\begin{aligned} \int _{\Omega }|u_0+w_\Omega (u_0)|^6\,\mathrm{d}x\le & {} \int _{\Omega }|u_0+w_0|^6\,\mathrm{d}x \le \liminf _{n\rightarrow \infty }\int _{\Omega }|u_n+w_\Omega (u_n)|^6\,\mathrm{d}x \\\le & {} \liminf _{n\rightarrow \infty } \int _{\Omega }|u_n+w_\Omega (u_0)|^6\,\mathrm{d}x = \int _{\Omega }|u_0+w_\Omega (u_0)|^6\,\mathrm{d}x. \end{aligned}$$

Hence all inequalities above must be equalities and it follows that $w_0=w_\Omega (u_0)$ and $w_\Omega (u_n)\rightarrow w_\Omega (u_0)$. $\square$

We shall need the following inequality:

Lemma 5.2

If $u\in W^6_0(\mathrm {curl};\Omega ){\setminus }\{0\}$, $w\in \mathcal {W}_\Omega$ and $t\ge 0$, then

$$\begin{aligned} J(u)\ge J(tu+w) -J'(u)\left[ \frac{t^2-1}{2}u+tw\right] . \end{aligned}$$

(5.4)

Moreover, strict inequality holds unless $t=1$ and $w=0$. ( $\Omega =\mathbb {R}^3$ admitted.)

Proof

The proof follows a similar argument as in [22, Proposition 4.1] and [23, Lemma 4.1]. We include it for the reader’s convenience. We show that

$$\begin{aligned} J(u)-J(tu+w) +J'(u)\left[ \frac{t^2-1}{2}u+tw\right] = \int _{\mathbb {R}^3}\varphi (t,x)\,\mathrm{d}x\ge 0, \end{aligned}$$

(5.5)

where

$$\begin{aligned} \varphi (t,x){:=} -\Big \langle |u|^{4}u,\frac{t^2-1}{2}u+tw\Big \rangle - \frac{1}{6}|u|^6 + \frac{1}{6}|tu+w|^6. \end{aligned}$$

An explicit computation using $\nabla \times w = 0$ shows that both sides of (5.5) are equal. Clearly, $\varphi (t,x)\ge 0$ if $u(x)=0$. So let $u(x)\ne 0$. It is easy to check that $\varphi (0,x) > 0$ and $\varphi (t,x)\rightarrow \infty$ as $t\rightarrow \infty$. Note that if $\partial _t\varphi (t_0,x)=0$ for some $t_0>0$, then either $\langle u,\,t_0u+w\rangle =0$ or $|u|=|t_0u+w|$. In the first case, substituting $-\langle u,w\rangle = t_0|u|^2$, we obtain $\varphi (t_0,x) = \big (\frac{t_0^2}{2}+\frac{1}{3}\big )|u|^6+\frac{1}{6}|t_0u+w|^6>0$. In the second case we have, using $-t_0\langle u, w\rangle = \frac{t_0^2-1}{2}|u|^2+\frac{1}{2}|w|^2$, that $\varphi (t_0,x) = \frac{1}{2}|u|^4|w|^2\ge 0$. Hence $\varphi (t,x)\ge 0$ for all $t\ge 0$ and the inequality is strict if $w\ne 0$. If $w=0$, then $\varphi (t,x)=\big (\frac{t^6}{6}-\frac{t^2}{2}+\frac{1}{3}\big )|u|^6>0$ provided $t\ne 1$. $\square$

Similarly as in (4.8) we introduce the set

$$\begin{aligned} {{\mathcal {N}}}_\Omega {:=}\Big \{u\in W^6_0(\mathrm {curl};\Omega ){\setminus }{{\mathcal {W}}}_\Omega : J'(u)u=0\hbox { and }J'(u)|_{{{\mathcal {W}}}_\Omega }=0\Big \}. \end{aligned}$$

(5.6)

Proof of Theorems 1.1 and 1.3

Since $tu+w(tu) = t(u+w(u))$ according to Lemma 4.2, we may assume without loss of generality that $u+w(u)\in {{\mathcal {N}}}$ in (5.2) and similarly, $u+w_\Omega (u)\in {{\mathcal {N}}}_\Omega$ in (5.3). According to Lemma 4.7,

$$\begin{aligned} \inf _{{{\mathcal {N}}}} J|_{W_0^6(\mathrm {curl};\Omega )} = \frac{1}{3}S_\mathrm {curl}(\Omega )^{\frac{3}{2}},\quad \inf _{{{\mathcal {N}}}_\Omega } J = \frac{1}{3}\overline{S}_\mathrm {curl}(\Omega )^{\frac{3}{2}},\quad \inf _{{\mathcal {N}}}J = \frac{1}{3}S_\mathrm {curl}^{\frac{3}{2}}. \end{aligned}$$

In view of Lemma 2.3, $\mathcal {W}_\Omega \subset {{\mathcal {W}}}$, hence we easily infer from (5.2), (5.3) that $S_\mathrm {curl}(\Omega )\ge \overline{S}_\mathrm {curl}(\Omega )$. As $W_0^6(\mathrm {curl};\Omega )\subset W_0^6(\mathrm {curl};\mathbb {R}^3)$, it follows that $S_\mathrm {curl}\le S_\mathrm {curl}(\Omega )$.

Next we show that $S_\mathrm {curl}(\Omega )\le S_\mathrm {curl}$. Let $u_0$ be a minimizer for J on ${{\mathcal {N}}}$ provided by Theorem 1.2(b) and find a sequence $(u_n)\subset {{\mathcal {C}}}_0^\infty (\mathbb {R}^3,\mathbb {R}^3)$ such that $u_n\rightarrow u_0$. We can decompose $u_n$ as $u_n=v_n+w_n$, $v_n\in {{\mathcal {V}}}$, $w_n\in {{\mathcal {W}}}$. Since $u_0=v_0+w(v_0)$ (recall $u_0\in {{\mathcal {N}}}$), $u_n=v_n+w_n\rightarrow u_0=v_0+w(v_0)$ and therefore $v_n\rightarrow v_0$, $w_n\rightarrow w(v_0)$. So $v_0\ne 0$ and $v_n$ are bounded away from 0 in $L^6(\mathbb {R}^3,\mathbb {R}^3)$. Assume without loss of generality that $0\in \Omega$. There exist $\lambda _n$ such that $\widetilde{u}_n$ given by $\widetilde{u}_n(x){:=}\lambda _n^{1/2}u_n(\lambda _nx)$ are supported in $\Omega$. Set $\widetilde{w}_n {:=} w(\widetilde{u}_n)\in {{\mathcal {W}}}$ and choose $t_n$ so that $t_n(\widetilde{u}_n+\widetilde{w}_n)\in {{\mathcal {N}}}$. Then

$$\begin{aligned} t_n^2 = \frac{|\nabla \times \widetilde{u}_n|_2}{|\widetilde{u}_n+\widetilde{w}_n|_6^3}. \end{aligned}$$

(5.7)

According to Lemma 4.6, $\Vert \widetilde{u}_n\Vert = \Vert u_n\Vert$ and $|\widetilde{u}_n+\widetilde{w}_n|_6 = |u_n+w(u_n)|_6 = |v_n+w(v_n)|_6$. As $(u_n)$ is bounded, so is $(\widetilde{u}_n)$ and as $|v_n+w(v_n)|_6\rightarrow |v_0+w(v_0)|_6$, $|\widetilde{u}_n+\widetilde{w}_n|_6$ is bounded away from 0. So $(t_n)$ is bounded. Moreover, $|\widetilde{w}_n|_6 = |w(u_n)|_6$ and therefore $(\widetilde{w}_n)$ is bounded. Since $J(\widetilde{u}_n)=J(u_n)\rightarrow \frac{1}{3}S_\mathrm {curl}^{3/2}$ and $\Vert J'(\widetilde{u}_n)\Vert = \Vert J'(u_n)\Vert \rightarrow 0$, it follows from Lemma 5.2 that

$$\begin{aligned} \frac{1}{3}S_\mathrm {curl}^{3/2} = \lim _{n\rightarrow \infty } J(\widetilde{u}_n)\ge & {} \lim _{n\rightarrow \infty } \left( J(t_n(\widetilde{u}_n+\widetilde{w}_n)) -J'(\widetilde{u}_n)\left[ \frac{t_n^2-1}{2}\widetilde{u}_n+t_n^2\widetilde{w}_n\right] \right) \\= & {} \lim _{n\rightarrow \infty } J(t_n(\widetilde{u}_n+\widetilde{w}_n)) \ge \frac{1}{3} S_\mathrm {curl}(\Omega )^{3/2}. \end{aligned}$$

The last inequality follows from Lemma 4.7 and the fact that $\widetilde{u}_n$ are as in (5.2), that is $\widetilde{u}_n\in W_0^6(\mathrm {curl};\Omega )$.

It remains to show that $\overline{S}_{\mathrm {curl}}(\Omega )\ge S$ if $(\Omega )$ is satisfied. But this follows by repeating the argument of Lemma 4.1 with obvious changes: $S_\mathrm {curl}$ should be replaced by $\overline{S}_{\mathrm {curl}}(\Omega )$, w(v) by $w_\Omega (v)$ and the domain of integration should be $\Omega$. $\square$

Remark 5.3

Let $\Omega \ne \mathbb {R}^3$ and suppose $S_\mathrm {curl}(\Omega )$ is attained by some u. Extend u by 0 outside $\Omega$. As $S_\mathrm {curl}(\Omega )=S_\mathrm {curl}$, u also solves (1.6) in $\mathbb {R}^3$, possibly after replacing u with $\alpha u$ for an appropriate $\alpha >0$. In particular, if $S_\mathrm {curl}(\Omega )$ were attained in a bounded $\Omega$, this would imply the existence of ground states in $\mathbb {R}^3$ which have compact support. To the best of our knowledge, there is no unique continuation principle which could rule out this possibility.

In view of this remark we expect that similarly as is the case for the Sobolev constant, $S_\mathrm {curl}$ is attained if and only if $\Omega =\mathbb {R}^3$. We leave this problem as a conjecture.

6 The Brezis–Nirenberg Type Problem and Proof of Theorem 1.4

Let $\lambda \le 0$. In this section $\Omega \subset \mathbb {R}^3$ is a fixed bounded domain satisfying $(\Omega )$ but $\lambda$ will be varying. Therefore we drop the subscript $\Omega$ from notation and replace it by $\lambda$ ($J_\lambda$, ${{\mathcal {N}}}_\lambda$ etc.). We also write ${{\mathcal {V}}},{{\mathcal {W}}}$ for ${{\mathcal {V}}}_\Omega , {{\mathcal {W}}}_\Omega$.

Recall from the introduction and Section 2.2 that the spectrum of the curl–curl operator in $H_0(\mathrm {curl};\Omega )$ consists of the eigenvalue $\lambda _0=0$ whose eigenspace is ${{\mathcal {W}}}$ and of a sequence of eigenvalues

$$\begin{aligned} 0<\lambda _1 \le \lambda _2\le \dots \le \lambda _k\rightarrow \infty , \end{aligned}$$

with finite multiplicities $m(\lambda _k)\in \mathbb {N}$. The eigenfunctions corresponding to different eigenvalues are $L^2$-orthogonal and those corresponding to $\lambda _k>0$ are in ${{\mathcal {V}}}$.

For $\lambda \le 0$ we find two closed and orthogonal subspaces ${{\mathcal {V}}}^+$ and ${\widetilde{{{\mathcal {V}}}}}$ of ${{\mathcal {V}}}$ such that the quadratic form $Q:{{\mathcal {V}}}\rightarrow \mathbb {R}$ given by

$$\begin{aligned} Q(v){:=}\int _\Omega (|\nabla \times v|^2+\lambda |v|^2)\, \mathrm{d}x \equiv \int _\Omega (|\nabla v|^2+\lambda |v|^2)\, \mathrm{d}x \end{aligned}$$

is positive definite on ${{\mathcal {V}}}^+$ and negative semidefinite on ${\widetilde{{{\mathcal {V}}}}}$ where $\mathrm {dim}\,{\widetilde{{{\mathcal {V}}}}}<\infty$. Writing $u=v+w=v^++\widetilde{v}+w\in {{\mathcal {V}}}^+\oplus \widetilde{{{\mathcal {V}}}}\oplus {{\mathcal {W}}}$, we have

$$\begin{aligned} Q(v)=Q(v^+)+Q(\widetilde{v}), \end{aligned}$$

and our functional $J_\lambda$ (see (1.9)) can be expressed as

$$\begin{aligned} J_\lambda (u) = \frac{1}{2}Q(v^+) + \frac{1}{2} Q(\widetilde{v}) + \frac{\lambda }{2}\int _{\Omega }|w|^2\,\mathrm{d}x - \frac{1}{6}|u|^6\,\mathrm{d}x. \end{aligned}$$

We shall use Theorem 3.1 with

$$\begin{aligned} F(x,u)=\frac{1}{6}|u|^6 -\frac{\lambda }{2}|u|^2. \end{aligned}$$

Here $\widetilde{{{\mathcal {W}}}} {:=} \widetilde{{{\mathcal {V}}}}\oplus {{\mathcal {W}}}$ (so $Z=\widetilde{{{\mathcal {V}}}}$ in the notation of Section 3) and $\widetilde{w} = \widetilde{v}+w$. ${{\mathcal {V}}}$, and hence ${{\mathcal {V}}}^+$, may be considered, after a proper extension, as closed subspaces of ${{\mathcal {D}}}^{1,2}(\mathbb {R}^3,\mathbb {R}^3)$. Indeed, let U be a bounded domain in $\mathbb {R}^3$, $U\supset \overline{\Omega }$. Since ${{\mathcal {V}}}\subset H^1(\Omega ,\mathbb {R}^3)$, each $v\in {{\mathcal {V}}}$ may be extended to $v'\in H^1_0(U,\mathbb {R}^3)$ such that $v'|_\Omega =v$. This extension is bounded as a mapping from ${{\mathcal {V}}}$ to $H^1_0(U,\mathbb {R}^3)$. Since

$$\begin{aligned} {{\mathcal {V}}}'{:=}\big \{v'\in H^1_0(U,\mathbb {R}^3): v'|_{\Omega }\in {{\mathcal {V}}}\big \} \end{aligned}$$

is a closed subspace of $H^1_0(U,\mathbb {R}^3)$, and hence of ${{\mathcal {D}}}^{1,2}(\mathbb {R}^3,\mathbb {R}^3)$, we can apply Theorem 3.1 with F as above and ${{\mathcal {V}}}^+$ replacing ${{\mathcal {V}}}$. The generalized Nehari manifold is now given by

$$\begin{aligned} {{\mathcal {N}}}_\lambda {:=}\{u\in W^6_0(\mathrm {curl};\Omega ){\setminus }({\widetilde{{{\mathcal {V}}}}}\oplus \mathcal {W}):\; J_\lambda '(u)|_{\mathbb {R}u\oplus {\widetilde{{{\mathcal {V}}}}}\oplus \mathcal {W}}=0\}. \end{aligned}$$

(6.1)

As in Section 4, also here it is not clear whether ${{\mathcal {N}}}_\lambda$ is of class ${{\mathcal {C}}}^1$. Setting $m_\lambda (v^+){:=}v^++\widetilde{w}(v^+)$ where $v^+\in {{\mathcal {V}}}^+$ and $\widetilde{w}(v^+) \equiv \widetilde{w}_\Omega (v^+)$ is the minimizer as in (3.2), we have

$$\begin{aligned} m_\lambda (v^+) {:=} t(v^+)(v^++\widetilde{w}(v^+)) \in {{\mathcal {N}}}_\lambda , \quad v^+\in {{\mathcal {V}}}^+{\setminus }\{0\} \end{aligned}$$

(cf. (4.10)) and $J_\lambda \circ m_\lambda$ is of class ${{\mathcal {C}}}^1$ on ${{\mathcal {S}}}^+$. Moreover, $m_\lambda |_{{{\mathcal {S}}}^+}$ is a homeomorphism between ${{\mathcal {S}}}^+$ and ${{\mathcal {N}}}_\lambda$. As in (4.13), we may find a Palais–Smale sequence $(v_n^+)\subset {{\mathcal {S}}}^+$ such that

$$\begin{aligned} (J_\lambda \circ m_\lambda )(v_n^+)\rightarrow \inf _{{{\mathcal {S}}}^+} J_\lambda \circ m_\lambda = c_\lambda \hbox { and }J_\lambda '(m_\lambda (v_n^+))\rightarrow 0 \end{aligned}$$

(6.2)

where

$$\begin{aligned} c_\lambda {:=}\inf _{{{\mathcal {N}}}_\lambda }J_\lambda . \end{aligned}$$

Note that

$$\begin{aligned} c_0 = \frac{1}{3} \overline{S}_\mathrm {curl}(\Omega )^{3/2} \ge \frac{1}{3} S^{3/2}. \end{aligned}$$

Lemma 6.1

Let $\lambda \in (-\lambda _\nu ,-\lambda _{\nu -1}]$ for some $\nu \ge 1$. There holds

$$\begin{aligned} c_\lambda \le \frac{1}{3}(\lambda +\lambda _\nu )^{3/2}|\Omega | \quad \text {and} \quad c_\lambda<c_0 \text { if } \lambda <-\lambda _\nu + \overline{S}_\mathrm {curl}(\Omega )|\Omega |^{-2/3}. \end{aligned}$$

Proof

The first inequality has been established in [23, Lemma 4.7]. However, for the reader’s convenience we include the argument. Let $e_\nu$ be an eigenvector corresponding to $\lambda _\nu$. Then $e_\nu \in {{\mathcal {V}}}^+$. Choose $t>0$, $\widetilde{v}\in \widetilde{{{\mathcal {V}}}}$ and $w\in {{\mathcal {W}}}$ so that $u= v+w=te_\nu +\widetilde{v}+w\in {{\mathcal {N}}}_\lambda$. Since $\lambda _k\le \lambda _\nu$ for $k<\nu$,

$$\begin{aligned} c_\lambda\le & {} J_\lambda (u) = \frac{1}{2}\int _{\Omega }|\nabla \times v|^2\,\mathrm{d}x +\frac{\lambda }{2}\int _{\Omega }|u|^2\,\mathrm{d}x -\frac{1}{6}\int _{\Omega }|u|^6\,\mathrm{d}x \\\le & {} \frac{\lambda _\nu }{2}\int _{\Omega }|v|^2\,\mathrm{d}x + \frac{\lambda }{2}\int _{\Omega }|u|^2\,\mathrm{d}x -\frac{1}{6}\int _{\Omega }|u|^6\,\mathrm{d}x \le \frac{\lambda +\lambda _\nu }{2}\int _{\Omega }|u|^2\,\mathrm{d}x -\frac{1}{6}\int _{\Omega }|u|^6\,\mathrm{d}x \\\le & {} \frac{\lambda +\lambda _\nu }{2}\,|\Omega |^{2/3}\left( \int _{\Omega }|u|^6\,\mathrm{d}x\right) ^{1/3} - \frac{1}{6}\int _{\Omega }|u|^6\,\mathrm{d}x \le \frac{1}{3}(\lambda +\lambda _\nu )^{3/2}|\Omega |. \end{aligned}$$

In the last step we have used the elementary inequality $\frac{A}{2}t^2-\frac{1}{6}t^6 \le \frac{1}{3}A^{3/2}$ ($A>0$).

Since $c_0 = \frac{1}{3}\overline{S}_\mathrm {curl}(\Omega )^{3/2}$, the second inequality follows immediately. $\square$

If $c_\lambda <c_0$, then in view of [23, Theorem 2.2 (a)] there is a Palais–Smale sequence $(u_n)\subset {{\mathcal {N}}}_\lambda$ such that $J_\lambda (u_n)\rightarrow c_\lambda >0$ and $u_n\rightharpoonup u_0\ne 0$ in $W^6_0(\mathrm {curl};\Omega )$. It has been unclear so far whether $u_0$ is a critical point of $J_\lambda$. Now we shall show using the concentration–compactness analysis from Section 3 that $u_0$ is not only a solution but even a ground state for (1.7). The following lemma plays a crucial role:

Lemma 6.2

If $(u_n)\subset {{\mathcal {N}}}_\lambda$ is bounded, then, passing to a subsequence, $u_n\rightarrow u_0$ in $L^2(\Omega ,\mathbb {R}^3)$ for some $u_0$.

Proof

Let $u_n = m_\lambda (v_n^+) = v_n^++\widetilde{w}(v_n^+)$. Since ${{\mathcal {V}}}^+$ and $\widetilde{{{\mathcal {W}}}}$ are complementary subspaces, $(v_n^+)$ is bounded in ${{\mathcal {V}}}^+$. So passing to a subsequence, $v_n^+\rightharpoonup v_0^+$ in ${{\mathcal {V}}}^+$, and $v_n^+\rightarrow v_0^+$ in $L^2(\Omega ,\mathbb {R}^3)$ and almost everywhere in $\Omega$. Hence, by Theorem 3.1, $\widetilde{w}(v_n^+)\rightarrow \widetilde{w}(v_0^+)$ in $L^2(\Omega ,\mathbb {R}^3)$, and therefore we also have that $u_n\rightarrow u_0$ there. $\square$

Lemma 6.3

(cf. [23, Lemma 4.6]) $J_\lambda$ is coercive on ${{\mathcal {N}}}_\lambda$.

Proof

Let $(u_n)$ be a sequence in ${{\mathcal {N}}}_\lambda$ such that $J_\lambda (u_n)\le d$. Then

$$\begin{aligned} d\ge J_\lambda (u_n) = J_\lambda (u_n) - \frac{1}{2} J_\lambda '(u_n)u_n = \frac{1}{3}\int _{\Omega }|u_n|^6\,\mathrm{d}x, \end{aligned}$$

hence $(u_n)$ is bounded in $L^6(\Omega ,\mathbb {R}^3)$, and therefore also in $L^2(\Omega ,\mathbb {R}^3)$. It follows that

$$\begin{aligned} d\ge J_\lambda (u_n) = \frac{1}{2}Q(v_n^+) + \frac{1}{2}Q(\widetilde{v}_n) + \frac{\lambda }{2}\int _{\Omega }|w_n|^2\,\mathrm{d}x - \frac{1}{6}\int _{\Omega }|u_n|^6\,\mathrm{d}x, \end{aligned}$$

where the last three terms are bounded (recall $\text {dim}\,\widetilde{{{\mathcal {V}}}}<\infty$). Hence also $(v_n^+)$ is bounded. $\square$

Let

$$\begin{aligned} N(u) {:=} |u|^4u. \end{aligned}$$

It is clear that $N: L^6(\Omega ,\mathbb {R}^3) \rightarrow L^{6/5}(\Omega ,\mathbb {R}^3)$. We shall need the following version of the Brezis–Lieb lemma:

Lemma 6.4

Suppose $(u_n)$ is bounded in $L^6(\Omega ,\mathbb {R}^3)$ and $u_n\rightarrow u$ almost everywhere in $\Omega$. Then

$$\begin{aligned} N(u_n)-N(u_n-u)\rightarrow N(u) \quad \text {in } L^{6/5}(\Omega ,\mathbb {R}^3) \text { as } n\rightarrow \infty . \end{aligned}$$

Proof

Since $N(u_n)-N(u_n-u)\rightarrow N(u)$ almost everywhere in $\Omega$ and $N(u_n)-N(u_n-u)$ is bounded in $L^{6/5}(\Omega ,\mathbb {R}^3)$, $N(u_n)-N(u_n-u)\rightharpoonup N(u)$. We claim that $|N(u_n)-N(u_n-u)|_{6/5}\rightarrow |N(u)|_{6/5}$. Using Vitali’s convergence theorem we obtain

$$\begin{aligned}&\int _{\Omega }\left| |u_n|^4u_n-|u_n-u|^4(u_n-u)\right| ^{6/5}\,\mathrm{d}x \\&\quad = \int _{\Omega }\int _0^1\frac{\mathrm{d}}{\mathrm{d}t}\left| |u_n+(t-1)u|^4(u_n+(t-1)u\right| ^{6/5}\,\mathrm{d}t\mathrm{d}x \\&\quad = \int _{\Omega }\int _0^1\frac{\mathrm{d}}{\mathrm{d}t}|u_n+(t-1)u|^6\,\mathrm{d}t\mathrm{d}x \\&\quad = 6\int _0^1\int _{\Omega }\langle |u_n+(t-1)u|^4(u_n+(t-1)u),\, u\rangle \,\mathrm{d}x\mathrm{d}t \\&\quad \rightarrow 6\int _0^1\int _{\Omega }t^5|u|^6\,\mathrm{d}x\mathrm{d}t = \int _{\Omega }|u|^6\,\mathrm{d}x. \end{aligned}$$

Hence $N(u_n)-N(u_n-u)$ converges strongly to N(u). $\square$

Lemma 6.5

Let $\beta <c_0$. Then $J_\lambda$ satisfies the $(PS)_\beta$-condition in ${{\mathcal {N}}}_\lambda$, that is if $(u_n)\subset {{\mathcal {N}}}_\lambda$, $J_\lambda (u_n)\rightarrow \beta$ and $J'_\lambda (u_n)\rightarrow 0$ as $n\rightarrow \infty$, then $u_n\rightarrow u_0\ne 0$ in $W_0^6(\mathrm {curl};\Omega )$ along a subsequence. In particular, $u_0$ is a nontrivial solution for (1.7)–(1.8).

Proof

Let $(u_n)$ be a $(PS)_\beta$-sequence such that $(u_n)\subset {{\mathcal {N}}}_\lambda$. According to Lemma 6.3, $(u_n)$ is bounded and we may assume $u_n\rightharpoonup u_0$ in $W_0^6(\mathrm {curl};\Omega )$. By Lemma 6.2, $u_n\rightarrow u_0$ in $L^{2}(\Omega ,\mathbb {R}^3)$ and hence also almost everywhere in $\Omega$ after passing to a subsequence if necessary. As in the proof of Theorem 1.2 in Section 4 we see that $J'_\lambda (u_0)=0$, that is $u_0$ is a solution for (1.7)–(1.8). According to the Brezis–Lieb lemma [9],

$$\begin{aligned} \lim _{n\rightarrow \infty }\Big (\int _{\Omega } |u_n|^{6} \,\mathrm{d}x - \int _{\Omega } |u_n-u_0|^{6} \,\mathrm{d}x\Big ) =\int _{\Omega } |u_0|^{6} \,\mathrm{d}x, \end{aligned}$$

hence

$$\begin{aligned} \lim _{n\rightarrow \infty }\big (J_\lambda (u_n)-J_\lambda (u_n-u_0)\big )=J_\lambda (u_0)\ge 0, \end{aligned}$$

(6.3)

and by Lemma 6.4,

$$\begin{aligned} \lim _{n\rightarrow \infty }\big (J_{\lambda }'(u_n)-J_{\lambda }'(u_n-u_0)\big )=J_{\lambda }'(u_0) = 0. \end{aligned}$$

(6.4)

Since $J_\lambda '(u_n)\rightarrow 0$ and $u_n\rightarrow u_0$ in $L^2(\Omega ,\mathbb {R}^3)$,

$$\begin{aligned} \lim _{n\rightarrow \infty } J_{0}'(u_n-u_0) = 0. \end{aligned}$$

(6.5)

Suppose $\liminf _{n\rightarrow \infty }\Vert u_n-u_0\Vert >0$. Since $\lim _{n\rightarrow \infty }J_{0}'(u_n-u_0)(u_n-u_0)=0$, we infer that

$$\begin{aligned} \liminf _{n\rightarrow \infty }|\nabla \times (u_n-u_0)|_2>0. \end{aligned}$$

Let $u_n-u_0=v_n+\widetilde{w}_n\in {{\mathcal {V}}}\oplus {{\mathcal {W}}}$ according to the Helmholtz decomposition in $W_0^6(\mathrm {curl};\Omega )$. If $v_n\rightarrow 0$ in $L^6(\Omega ,\mathbb {R}^3)$, then by (6.5) we have $J_{0}'(u_n-u_0)v_n\rightarrow 0$, thus

$$\begin{aligned}&|\nabla \times (u_n-u_0)|_2^2 = |\nabla \times v_n|_2^2=J_{0}'(u_n-u_0)v_n\\&+\int _{\Omega }\langle |u_n-u_0|^4(u_n-u_0),v_n\rangle \,\mathrm{d}x \rightarrow 0 \end{aligned}$$

as $n\rightarrow \infty$, which is a contradiction. Therefore $|v_n|_6$ is bounded away from 0. Put $w_n{:=} w(u_n-u_0)\in {{\mathcal {W}}}$. Then $(w_n)$ is bounded and since $u_n-u_0+w_n=v_n+w(v_n)\in {{\mathcal {V}}}\oplus {{\mathcal {W}}}$, $|u_n-u_0+w_n|_6$ is bounded away from 0. Choose $t_n$ so that $t_n(u_n-u_0+w_n)\in {{\mathcal {N}}}_0$ (${{\mathcal {N}}}_0\equiv {{\mathcal {N}}}_\Omega$ in the notation of Section 5). As in (5.7), we have

$$\begin{aligned} t_n^2 = \frac{|\nabla \times (u_n-u_0)|_2}{|u_n-u_0+w_n|_6^3}, \end{aligned}$$

so $(t_n)$ is bounded. Using Lemma 5.2, as in the proof of Theorems 1.1 and 1.3 we get

$$\begin{aligned} J_{0}(u_n-u_0)\ge J_{0}(t_n(u_n-u_0+w_n))-J_{0}'(u_n-u_0)\Big [\frac{t_n^2-1}{2}(u_n-u_0)+t_n^2w_n\Big ], \end{aligned}$$

so by (6.5) and since $u_n\rightarrow u_0$ in $L^2(\Omega ,\mathbb {R}^3)$,

$$\begin{aligned} \beta = \lim _{n\rightarrow \infty }J_\lambda (u_n-u_0)= \lim _{n\rightarrow \infty }J_{0}(u_n-u_0)\ge \lim _{n\rightarrow \infty }J_{0}(t_n(u_n-u_0+w_n))\ge c_0, \end{aligned}$$

which is a contradiction. Therefore, passing to a subsequence, $u_n\rightarrow u_0$. Since $u_0\in {{\mathcal {N}}}_\lambda$, $u_0\ne 0$. $\square$

Proof of Theorem 1.4

(a) It follows from (6.2) and Lemma 6.5 that if $c_\lambda <c_0$, then $c_\lambda$ is attained and hence there exists a ground state solution. By Lemma 6.1, this inequality is satisfied whenever $\lambda \le \lambda _{\nu -1}$ and $\lambda \in (-\lambda _\nu ,-\lambda _\nu + \overline{S}_\mathrm {curl}(\Omega )|\Omega |^{-2/3})$.

In view of [23, Theorem 2.2(b)], the function $(-\lambda _{\nu },-\lambda _{\nu -1}]\ni \lambda \mapsto c_\lambda \in (0,+\infty )$ is non-decreasing, continuous and $c_\lambda \rightarrow 0$ as $\lambda \rightarrow -\lambda _\nu ^-$, and if $c_{\mu _1}=c_{\mu _2}$ for some $-\lambda _{\nu }<\mu _1<\mu _2\le -\lambda _{\nu -1}$, then $c_\lambda$ is not attained for $\lambda \in (\mu _1,\mu _2]$. Hence (b) and (c) follow.

(d) Since $J_\lambda$ is even and, by Lemma 6.5, satisfies the Palais–Smale condition in ${{\mathcal {N}}}_\lambda$ at any level below $c_0$, then, in view of [23, Theorem 3.2(c)], $J_\lambda$ has at least $m({{\mathcal {N}}}_\lambda ,c_0)$ pairs of critical points $\pm u$ such that $u\ne 0$ and $c_\lambda \le J_\lambda (u)<c_0$ where

$$\begin{aligned} m({{\mathcal {N}}}_\lambda ,c_0){:=}\sup \{\gamma (J_\lambda ^{-1}((0,\beta ])\cap {{\mathcal {N}}}_\lambda ): \beta <c_0\} \end{aligned}$$

(6.6)

and $\gamma$ is the Krasnoselskii genus [30]. This is a consequence of the standard fact that if

$$\begin{aligned} \beta _k {:=} \inf \{\beta \in \mathbb {R}: \gamma (J_\lambda ^{-1}((0,\beta ])\cap {{\mathcal {N}}}_\lambda ) \ge k\}, \end{aligned}$$

then there are at least as many pairs of critical points as the number of k for which $(PS)_{\beta _k}$ holds, see for example [30].

In order to complete the proof we show that

$$\begin{aligned} m({{\mathcal {N}}}_\lambda ,c_0)\ge \widetilde{M}(\lambda ) {:=} \#\Big \{k: -\lambda _k<\lambda <-\lambda _k +\frac{1}{3}\overline{S}_\mathrm {curl}(\Omega )|\Omega |^{-\frac{2}{3}}\Big \}. \end{aligned}$$

Let

$$\begin{aligned} A(\lambda ){:=}\big \{k\ge 1: -\lambda _k< \lambda <-\lambda _k +\frac{1}{3}\overline{S}_\mathrm {curl}(\Omega )|\Omega |^{-\frac{2}{3}}\hbox { and }\lambda _k>\lambda _{k-1}\} \end{aligned}$$

and observe that

$$\begin{aligned} \widetilde{M}(\lambda )=\sum _{k\in A(\lambda )}m(\lambda _k), \end{aligned}$$

where $m(\lambda _k)$ stands for the multiplicity of $\lambda _k$. For $k\in A(\lambda )$, let $\mathcal {V}(\lambda _k)$ denote the eigenspace corresponding to $\lambda _k$. Then $\text {dim}\,\mathcal {V}(\lambda _k)=m(\lambda _k)$. Let $S(\lambda )$ be the unit sphere in $\bigoplus _{k\in A(\lambda )}\mathcal {V}(\lambda _k)\subset \mathcal {V}^{+}$. Recall that $m_{\lambda }|_{{{\mathcal {S}}}^+}$ is a homeomorphism from ${{\mathcal {S}}}^+$ to ${{\mathcal {N}}}_\lambda$. Since $J_\lambda$ is even, $m_{\lambda }$ is odd. Similarly as in Lemma 6.1 we show that for $u\in S(\lambda )$

$$\begin{aligned} J_\lambda (m_\lambda (u))\le \max _{k\in A(\lambda )}\frac{1}{3}(\lambda +\lambda _{k})^{\frac{3}{2}}|\Omega |{=:}\beta \end{aligned}$$

and thus $m_\lambda (S(\lambda ))\subset J_\lambda ^{-1}((0,\beta ])\cap {{\mathcal {N}}}_\lambda$. Hence

$$\begin{aligned} \gamma (J_\lambda ^{-1}(0,\beta ]\cap {{\mathcal {N}}}_\lambda )\ge \gamma (S(\lambda ))=\widetilde{m}_\lambda . \end{aligned}$$

Since $\lambda <-\lambda _k +\frac{1}{3}\overline{S}_\mathrm {curl}(\Omega )|\Omega |^{-\frac{2}{3}}$ (cf. Lemma 6.1), we have $\beta <c_0$ and it follows that $m({{\mathcal {N}}}_\lambda ,c_0)\ge \widetilde{M}(\lambda )$ which completes the proof. $\square$

7 Open Problems

In this section we state some open problems. Some of them have already been mentioned earlier.

• (P1) Does there exist a ground state solution u whose support is a proper subset of $\mathbb {R}^3$? In particular, can a ground state have compact support?

• (P2) Can one find an explicit expression for a ground state? Or at least, what can be said about the decay properties of ground states? If they are the same as for the Aubin–Talenti instantons, then one could hopefully retrieve the formulas in the middle of p. 35 in [36] which could be useful when looking for ground states for (1.6) with the right-hand side $|u|^4u+g(x,u)$ where g is a monotone lower order term.

• (P3) Do the groud state solutions to (1.6) have any symmetry properties? How regular are they?

• (P4) If $\Omega$ is a bounded domain which is neither convex nor has ${{\mathcal {C}}}^{1,1}$ boundary, then ${{\mathcal {V}}}\subset H^s(\Omega ,\mathbb {R}^3)$ where $s\in [1/2,1]$ and s may be strictly less than 1, see Section 2.2 and [12]. Note that the critical exponent for $H^s$ is $6/(3-2s)<6$ if $s<1$. Do the results of Theorem 1.4 remain valid (with the same right-hand side)? Here the boundary condition (1.8) should be understood in a generalized sense, that is u should be in $W_0^6(\mathrm {curl};\Omega )$.

• (P5) Can the inequality $S_\mathrm {curl}\ge \overline{S}_\mathrm {curl}(\Omega )\ge S$ be sharpened? Do there exist domains as in (P4) for which $\overline{S}_\mathrm {curl}(\Omega )<S$?