Existence of a mountain pass solution for a nonlocal fractional $(p, q)$-Laplacian problem

Abstract

Here, a nonlocal nonlinear operator known as the fractional $(p,q)$-Laplacian is considered. The existence of a mountain pass solution is proved via critical point theory and variational methods. To this aim, the well-known theorem on the construction of the critical set of functionals with a weak compactness condition is applied.

1 Introduction

The quasilinear operator $(p,q)$-Laplacian has been used to model steady-state solutions of reaction–diffusion problems arising in biophysics, in plasma physics and in the study of chemical reactions. These problems appear, for example, in a general reaction–diffusion system:

$$u_{t}=-\operatorname{div}\bigl[D(u)\nabla u\bigr]+h(x,u), $$

where $D(u) =|\nabla u|^{p-2}+|\nabla u|^{q-2}$ is the diffusion coefficient, the function u describes a concentration and the reaction term $h(x, u)$ has a polynomial form with respect to the concentration u. The differential operator $\Delta_{p} + \Delta_{q}$ is known as the $(p, q)$-Laplacian operator, if $p\neq q$, where $\Delta_{j}$, $j>1$ denotes the j-Laplacian and is defined by $\Delta_{j} u:=\operatorname{div}(|\nabla u|^{j-2} \nabla u)$. It is not homogeneous, thus some technical difficulties arise in applying the usual methods of the theory of elliptic equations, for further details see [1, 6, 11, 12]. When $p=q$, we obtain the p-Laplacian operator that was extensively studied by many authors; see [8, 14, 21–26, 28, 33–37]. Moreover, Marano et al. [27] studied the existence of solutions for the nonlinear elliptic problem of $(p, q)$-Laplacian type

$$ \textstyle\begin{cases} -\Delta_{p}u- \mu\Delta_{q}u= f(x, u), & x \in\Omega,\\ u=0 ,& x\in\partial\Omega, \end{cases} $$

where $\Omega\subset\mathbb{R}^{N}$ ($N\geq1$) is a bounded domain with boundary of class $C^{2}$, $1< q \leq p< N$, $\mu\in\mathbb{R}_{0}^{+}$.

Recently, a great attention has been focused on the study of problems involving fractional operators of elliptic type, both in pure mathematical research and applications in real world, such as optimization, finance, population dynamics, minimal surfaces, water waves, game theory and so on. The literature on fractional operators and their applications to partially differential equations is quite large; for more details, see [3, 29, 35, 42]. In [5] authors study a new fractional Sobolev space and applications to nonlocal variational problems with variable exponent.

Using the Leray–Schauder nonlinear alternative, Qiu et al. [30] proved the existence of solutions for the fractional p-Laplacian problem

$$ \textstyle\begin{cases} (-\Delta)_{p}^{r} u= f(x, u), & x \in\Omega,\\ u=0, & x \in\mathbb{R}^{N} \setminus\Omega, \end{cases} $$

where Ω is an open bounded domain in $\mathbb{R}^{N}$ with the Lipschitz boundary, $1< p<\frac{N}{r}$, $0< r<1$ and $f: \Omega\times \mathbb{R} \rightarrow \mathbb{R}$ is a Carathéodory function. Also, $(-\Delta)_{p}^{r}$ is a nonlocal nonlinear operator known as the fractional p-Laplacian problem and is defined for u smooth enough by

$$ (-\Delta)_{p}^{r}u(x)=2\lim_{\varepsilon\searrow0} \int_{\mathbb {R}^{N} \setminus B_{\epsilon}(x)} \frac { \vert u(x)-u(y) \vert ^{p-2}(u(x)-u(y))}{ \vert x-y \vert ^{N+rp}}\,dy, $$

for every $x \in\mathbb{R}^{N}$, where $B_{\epsilon}(x)$ is the ball in $\mathbb{R}^{N}$ centered at $x \in\mathbb{R}^{N}$ and with radius $\epsilon>0$. We refer to [10] for the details and history of this operator. For more details on nonlocal operators we refer to [39], where many properties of these operators are investigated. In [13] existence and multiplicity results for fractional $(p, q)$-Laplacian type equations in $\mathbb{R}^{N}$ have been studied. Very recently, Bhakta et al. [7] studied the existence of infinitely many nontrivial weak solutions of the following fractional $(p, q)$-Laplacian equation:

$$ \textstyle\begin{cases} (-\Delta)_{p}^{r}u+(-\Delta)_{q}^{s}u=\theta V(x) \vert u \vert ^{m-2}u+ \vert u \vert ^{p_{r}^{*}-2}u+\lambda f(x,u),& x \in\Omega,\\ u=0, & x\in\mathbb{R}^{N} \setminus\Omega, \end{cases} $$

where $\Omega\subset\mathbb{R}^{N}$ is a smooth bounded domain, $\lambda, \theta>0$, $0< s< r<1<m<q<p<\frac{N}{r}$, the fractional critical exponent $p_{r}^{*}$ is defined by

$$ p_{r}^{*}:= \textstyle\begin{cases} \frac{Np}{N-rp} & \text{if } rp< N,\\ \infty& \text{if } rp \geq N. \end{cases} $$

Moreover, the functions V and f satisfy in the following conditions:

$(A_{1})$:

$V \in L^{\infty}(\Omega)$ and there exist $\sigma, \eta>0$ such that $V(x)>\sigma>0$ for every $x \in\Omega$ and $\int_{\Omega}V(x) |u|^{m} \,dx \leq\eta\|u\|_{s,m}^{m}$.

$(A_{2})$:

$|f(x,t)|\leq a_{1} |t|^{\alpha-1}+a_{2} |t|^{\beta-1}$ for all $x \in\Omega$, $t \in\mathbb{R}$, $a_{1}, a_{2} >0$ and $1<\alpha, \beta<p_{r}^{*}$.

$(A_{3})$:

There exist $a_{3}>0$ and $l \in(1, p)$ such that $f(x,t)t -p_{r}^{*}F(x,t) \geq-a_{3} |t|^{l}$, for all $x \in\Omega$, $t \in \mathbb{R}$ where $F(x,t)=\int_{0}^{t} f(x, \tau) \,d \tau$.

$(A_{4})$:

$f(x,t)>0$ for all $x \in\Omega$, $t \in\mathbb{R}^{+}$ and $f(x,t)=-f(x,-t)$ for all $x \in\Omega$, $t \in\mathbb{R}$.

Notice that, since f is odd in t, the energy functional I is even. It is worth noticing that without any symmetry assumption multiplicity results for some problems are not known, for example consider the following problem:

$$ \textstyle\begin{cases} -\Delta u=\lambda u-g(u) & \text{in } \Omega, \\ u=0 & \text{on } \partial\Omega, \end{cases} $$

(1.1)

where $\Omega\subset\mathbb{R}^{N}$ is a bounded domain, $\lambda\in \mathbb{R}$, $g: \mathbb{R} \rightarrow\mathbb{R}$ is a continuous function and $0<\lambda_{1}<\lambda_{2}<\cdots$ denote the eigenvalues of −Δ. If g is odd function, then for $\lambda_{k}<\lambda <\lambda_{k+1}$ problem (1.1) has at least k pairs of distinct nontrivial solutions. If g is not odd, multiplicity results fail (see [41, page 147]).

However, Dancer [15] showed that in general even for large enough λ one can expect no more than four nontrivial solutions. If $\lambda=1$, for odd nonlinearities Ambrosetti and Rabinowitz [2] and Rabinowitz [31] studied this kind of problems. In this work, the methods depend on the use of Lusternik–Schnirelman theory or rather on the concept of the genus for symmetric sets. Hence, the fact that energy functional is even is essential for applying these techniques. A natural and open question is to know whether the infinitely many solutions hold under perturbations of the odd equation. For more details of perturbations, we refer the interested reader to [4].

In this paper, we study a quasilinear problem, that is, a fractional $(p, q)$-Laplacian elliptic problem as

$$ \textstyle\begin{cases} (-\Delta)_{p}^{r}u+\gamma(-\Delta)_{q}^{s}u=\lambda \vert u \vert ^{p-2}u+ f(x,u),& x \in\Omega,\\ u=0, & x\in\mathbb{R}^{N} \setminus\Omega, \end{cases} $$

(1.2)

where $\Omega\subset\mathbb{R}^{N}$ is a bounded Lipschitz domain with $N \geq2$, $0< s< r<1$, $1< q< p< \frac{N}{r}$, $\gamma>0$, $\lambda< \lambda_{1}$ where the value of $\lambda_{1}$ is given in Sect. 3. In addition, $f: {\Omega} \times\mathbb{R}\rightarrow \mathbb{R}$ is a Carathéodory function with regularity assumptions on Ω. Due to nonlocality of the operator $(-\Delta)_{p}^{r}$, in this kind of problems, Dirichlet boundary condition $u=0$ given in $\mathbb{R}^{N} \setminus\Omega$ and not simply on ∂Ω. In this case, more careful analysis is needed.

This paper is motivated by results of Servadei et al. [39], where they proved the existence of mountain pass type solutions in a Hilbert space for

$$ \textstyle\begin{cases} \mathcal{L}_{k} u =\lambda u+ f(x, u), & x \in\Omega,\\ u=0, & x \in\mathbb{R}^{N} \setminus\Omega, \end{cases} $$

where $\lambda\in\mathbb{R}$, $\mathcal{L}_{k}$ is a nonlocal operator and f satisfies superlinear and subcritical growth conditions at zero and at infinity.

Here, we prove that problem (1.2) has a mountain pass type solution in the fractional Sobolev space which is not a Hilbert space. To this aim, some definitions and propositions are recalled in the sequel.

The norm in $L^{p}(\mathbb{R}^{N})$ is

$$ \Vert u \Vert _{L^{p}(\mathbb{R}^{N})}= \biggl( \int_{\mathbb{R}^{N}} \bigl\vert u(x) \bigr\vert ^{p} \,dx \biggr)^{\frac{1}{p}}. $$

By [16], for any $p \in(1, \infty)$ and $r \in(0, 1)$, the fractional Sobolev space $W^{r,p}(\mathbb{R}^{N})$ can be defined as

$$ \begin{gathered} W^{r,p}(\mathbb{R}^{N})\\\quad:= \biggl\{ u \in L^{p}\bigl(\mathbb{R}^{N}\bigr): u \text{ is measurable and } \int _{\mathbb{R}^{2N}} \frac{ \vert u(x)-u(y) \vert ^{p}}{ \vert x-y \vert ^{N+rp}}\,dx \,dy < \infty\biggr\} , \end{gathered} $$

with the norm

$$ \Vert u \Vert _{W^{r,p}(\mathbb{R}^{N})}:= \biggl( \int_{\mathbb{R}^{N}} \bigl\vert u(x) \bigr\vert ^{p} \,dx+ \int_{\mathbb{R}^{2N}} \frac { \vert u(x)-u(y) \vert ^{p}}{ \vert x-y \vert ^{N+rp}}\,dx \,dy \biggr)^{\frac{1}{p}}, $$

where $\frac{1}{|x-y|^{N+rp}}$ is the so-called singular kernel. Let $\Omega\subset\mathbb{R}^{N}$ be a bounded Lipschitz domain with $N \geq2$. We consider the standard fractional Sobolev space

$$W^{r,p}(\Omega):=\biggl\{ u \in L^{p}(\Omega): \frac { \vert u(x)-u(y) \vert }{ \vert x-y \vert ^{\frac{N}{p}+r}}\in L^{p}(\Omega\times\Omega)\biggr\} , $$

with respect to the localized norm

$$ \Vert u \Vert _{W^{r,p}(\Omega)}:= \biggl( \int_{\Omega}\bigl\vert u(x) \bigr\vert ^{p} \,dx + \int _{\Omega\times\Omega} \frac{ \vert u(x)-u(y) \vert ^{p}}{ \vert x-y \vert ^{N+rp}}\,dx \,dy \biggr)^{\frac{1}{p}}. $$

The closure of $C_{0}^{\infty}(\Omega)$ in $W^{r,p}(\Omega)$ is denoted by $W_{0}^{r,p}(\Omega)$.

Remark 1.1

Notice that $C_{0}^{\infty}(\Omega)$ is dense in $W_{0}^{r,p}(\Omega)$. Specially, restriction to Ω of any function in $W_{0}^{r,p}(\Omega)$ belongs to the closure of $C_{0}^{\infty}(\Omega)$ in $W^{r,p}(\Omega)$. From [8], if $rp \leq1$ for the seminorm localized on $\Omega\times\Omega$ Poincaré inequality with $\int_{\Omega}|u(x)|^{p} \,dx$ is not true.

We set $\mathcal{Q}:=\mathbb{R}^{2N}\setminus (\Omega^{c} \times\Omega^{c} )$, where $\Omega^{c} =\mathbb{R}^{N} \setminus\Omega$ and define

$$ \begin{gathered} X^{r, p}(\Omega)\\\quad:= \biggl\{ u: \mathbb{R}^{N} \rightarrow\mathbb{R} \text{ measurable}: u|_{\Omega}\in L^{p}(\Omega) \text{ and } \int_{\mathcal{Q}} \frac { \vert u(x)-u(y) \vert ^{p}}{ \vert x-y \vert ^{N+rp}}\,dx \,dy< \infty\biggr\} . \end{gathered} $$

The space $X^{r, p}(\Omega)$ is endowed with the following norm:

$$ \Vert u \Vert _{X^{r,p}(\Omega)}:= \biggl( \int_{\Omega}\bigl\vert u(x) \bigr\vert ^{p} \,dx+ \int _{\mathcal{Q}} \frac{ \vert u(x)-u(y) \vert ^{p}}{ \vert x-y \vert ^{N+rp}}\,dx \,dy \biggr)^{\frac{1}{p}}. $$

It is worth noticing that in general $W^{r, p}(\Omega)$ is not the same as $X^{r, p}(\Omega)$ as $\Omega\times\Omega$ is strictly contained in $\mathcal{Q}$. Now, we define

$$ X_{0}^{r, p}(\Omega)=\bigl\{ u \in X^{r, p}(\Omega): u(x)=0 \text{ a.e. in } \mathbb{R}^{N} \setminus\Omega\bigr\} , $$

(1.3)

or equivalently as $\overline{C_{0}^{\infty}(\Omega)}^{X^{r,p}(\Omega)}$. Since $u=0$ in $\mathbb{R}^{N} \setminus\Omega$, the Gagliardo norm can be defined as follows:

$$ \begin{aligned} \Vert u \Vert _{X_{0}^{r,p}(\Omega)}&:= \biggl( \int_{\mathbb{R}^{2N}} \frac { \vert u(y)-u(x) \vert ^{p}}{ \vert y-x \vert ^{N+rp}}\,dx \,dy \biggr)^{\frac{1}{p}} \\ &= \biggl( \int_{\mathbb{R}^{2N}} \frac{ \vert u(x+h)-u(x) \vert ^{p}}{ \vert h \vert ^{N+rp}}\,dx \,dh \biggr)^{\frac{1}{p}}, \end{aligned} $$

for all measurable functions $u:\mathbb{R}^{N} \longrightarrow\mathbb{R}$. Fractional Sobolev-type spaces are also called Aronszajn, Gagliardo or Slobodeckij spaces, by the names of the ones who first introduced them [40].

Notations: For simplicity of notations, we set $\|\cdot\|_{p}:=\|\cdot\| _{L^{p}(\Omega)}$, $X:=X_{0}^{r,p}(\Omega)$, $\|\cdot\|_{r, p}:=\|\cdot\| _{X_{0}^{r,p}(\Omega)}$ and $X^{*}:=X_{0}^{-r,p' }(\Omega)$ the dual space of X, where $\frac{1}{p}+\frac{1}{p' }=1$.

The natural function space to study fractional $(p, q)$-Laplacian problems in the nonlocal framework is fractional Sobolev space X. It is easy to check that $(X, \| \cdot \|_{r, p})$ is a uniformly convex Banach space.

Proposition 1.1

([7, Lemma 2.2])

Let $1 < q \leq p \leq\infty$and $0< s< r<1$. Assume Ω is a smooth bounded domain in $\mathbb{R}^{N}$, where $N>rp$. Then

$$ \Vert u \Vert _{s, q} \leq C \Vert u \Vert _{r,p},\quad \textit{for all } u \in X_{0}^{r, p}(\Omega), $$

for some suitable constant $C = C(|\Omega|,N, r, s, p, q) >0$. In particular,

$$ X_{0}^{r, p}(\Omega) \subseteq X_{0}^{s, q}( \Omega). $$

Here we consider the following problem:

$$ \begin{gathered}[b] \int_{\mathbb{R}^{2N}}\frac{ ( \vert u(x)-u(y) \vert ^{p-2}(u(x)-u(y)) ) (v(x)-v(y) )}{ \vert x-y \vert ^{N+rp}}\,dx \,dy \\ \qquad{}+\gamma \int_{\mathbb{R}^{2N}}\frac{ ( \vert u(x)-u(y) \vert ^{q-2}(u(x)-u(y)) ) (v(x)-v(y) )}{ \vert x-y \vert ^{N+sq}}\,dx \,dy \\ \quad=\lambda \int_{\Omega}\bigl\vert u(x) \bigr\vert ^{p-2}u(x)v(x) \,dx+ \int_{\Omega}f\bigl(x, u(x)\bigr)v(x) \,dx, \end{gathered} $$

(1.4)

for all $u, v \in X$. It is clear that (1.4) is the weak formulation of (1.2). The function F is the primitive of f and it is defined as follows:

$$ F(x, \xi):= \int_{0}^{\xi}f(x, t)\,dt, $$

(1.5)

for every $(x, \xi) \in\Omega\times\mathbb{R}$, then the energy functional $I_{\lambda}: X \rightarrow\mathbb{R}$ associated with (1.2) is defined by

$$I_{\lambda}(u):=\frac{1}{p} \Vert u \Vert _{r, p}^{p}+\frac{\gamma}{q} \Vert u \Vert _{s,q}^{q}-\frac{\lambda}{p} \int_{\Omega}\bigl\vert u(x) \bigr\vert ^{p} \,dx- \int_{\Omega}F\bigl(x, u(x)\bigr) \,dx, $$

for all $u \in X$. Also we assume that the Carathéodory function $f: \Omega\times \mathbb{R} \rightarrow\mathbb{R}$ satisfies the following conditions:

$(f_{1})$:

$|f(x, t)| \leq a_{1}+a_{2}|t|^{b-1}$, a.e. $x \in \Omega$, $t \in\mathbb{R}$, where $a_{1}, a_{2}>0$ and $b \in(p, p_{r}^{*})$.

$(f_{2})$:

$\lim_{|t|\rightarrow0}\frac{f(x, t)}{|t|^{p-1}}=0$ uniformly in $x \in\Omega$.

$(f_{3})$:

$0< \mu F(x, t) \leq t f(x, t)$, a.e. $x \in \Omega$, $t \in\mathbb{R}$, $|t|\geq\rho$, where $\mu>p$ and $\rho>0$.

Here is the main result of this paper.

Theorem 1.1

Let Ω be an open bounded set of $\mathbb{R}^{N}$with Lipschitz boundary, $0< s< r<1<q<p< \frac{N}{r}$and $\lambda<\lambda_{1}$, where the value of $\lambda_{1}$is given in Sect. 3. Consider the following equation:

$$ \begin{gathered}[b] \int_{\mathbb{R}^{2N}}\frac{ ( \vert u(x)-u(y) \vert ^{p-2}(u(x)-u(y)) ) (v(x)-v(y) )}{ \vert x-y \vert ^{N+rp}}\,dx \,dy \\ \qquad{}+\gamma \int_{\mathbb{R}^{2N}}\frac{ ( \vert u(x)-u(y) \vert ^{q-2}(u(x)-u(y)) ) (v(x)-v(y) )}{ \vert x-y \vert ^{N+sq}}\,dx \,dy \\ \quad=\lambda \int_{\Omega}\bigl\vert u(x) \bigr\vert ^{p-2}u(x)v(x) \,dx+ \int_{\Omega}f\bigl(x, u(x)\bigr)v(x) \,dx, \end{gathered} $$

(1.6)

for any $v \in X$. If f is a Carathéodory function verifying $(f_{1})$–$(f_{3})$, then for every $\gamma>0$problem (1.6) admits a mountain pass type solution $u \in X$which is not identically zero.

The paper is organized as follows: Sect. 2, is devoted to proving some regularities on the space X and the function f. In Sect. 3, we recall some properties of fractional eigenvalue problems. Finally, in Sect. 4, we show by some propositions that the conditions of the mountain pass theorem hold and we prove the existence of a solution of problem (1.2).

2 Preliminaries

In this section, we start with some preliminary lemmas and results on X and the functional f.

Lemma 2.1

([16, Theorem 6.5])

Let $r \in(0,1)$and $p \in[1, +\infty)$be such that $rp< N$. Then for every $W^{r,p}(\mathbb{R}^{N})$

$$ \Vert u \Vert _{L^{p_{r}^{*}}(\mathbb{R}^{N})}^{p} \leq c \int_{\mathbb{R}^{2N}} \frac{ \vert u(x)-u(y) \vert ^{p}}{ \vert x-y \vert ^{N+rp}}\,dx \,dy, $$

where $c=c(N, r, p)>0$. Consequently, $W^{r,p}(\mathbb{R}^{N})$is continuously embedded in $L^{k}(\mathbb{R}^{N})$for any $k \in[p, p_{r}^{*}]$and the embedding $W^{r,p}(\mathbb{R}^{N})$into $L^{k}(\mathbb {R}^{N})$is compact for every $k \in[p, p_{r}^{*}[$.

Remark 2.1

[19, page 159] Let $0< r<1$ and $p>1$. Then the embedding $X \hookrightarrow L^{k}(\Omega)$ is continuous for any $k \in[1, p_{r}^{*}]$ if $N>rp$, for any $k \in[1, \infty[$ if $N=rp$ and into $L^{\infty}(\Omega)$ if $N< rp$. The embedding is compact for any $k \in[1, p_{r}^{*}[$ if $N \geq rp$ and into $L^{\infty}(\Omega)$ if $N< rp$.

Now, we study some properties of functional f (see Lemmas 2.2 and 2.3). The proofs of Lemma 2.2 and Lemma 2.3 are similar to [38, Lemma 3 and Lemma 4], respectively.

Lemma 2.2

Let $f:\Omega\times\mathbb{R} \rightarrow\mathbb{R}$be a Carathéodory function such that conditions $(f_{1})$and $(f_{2})$hold. Then for any $\epsilon>0$there exists $\delta=\delta(\epsilon)$such that a.e. $x \in\Omega$and for any $t \in\mathbb{R}$

$$ \bigl\vert f(x, t) \bigr\vert \leq p \epsilon \vert t \vert ^{p-1}+b \delta(\epsilon) \vert t \vert ^{b-1}, $$

(2.1)

and hence, integrating (2.1) we get

$$ \bigl\vert F(x, t) \bigr\vert \leq\epsilon \vert t \vert ^{p}+\delta(\epsilon) \vert t \vert ^{b}. $$

(2.2)

Lemma 2.3

Suppose $f:\Omega\times\mathbb{R} \rightarrow\mathbb{R}$is a Carathéodory function such that condition $(f_{3})$holds. Then there exist two positive measurable functions $m=m(x)$and $M=M(x)$such that a.e. $x \in\Omega$and for any $t \in\mathbb{R}$

$$ F(x, t)\geq m(x) \vert t \vert ^{\mu}-M(x), $$

(2.3)

where F is defined as (1.5). Moreover, if the function f satisfies conditions $(f_{1})$and $(f_{2})$, then the functions $m, M \in L^{\infty}(\Omega)$.

3 Fractional eigenvalue problems

In this section, we consider the generalized eigenvalue problem as follows:

$$ \textstyle\begin{cases} (-\Delta)_{p}^{r}u+\gamma(-\Delta)_{q}^{s}u=\lambda \vert u \vert ^{j-2}u, & x \in\Omega,\\ u=0, & x\in\mathbb{R}^{N} \setminus\Omega, \end{cases} $$

(3.1)

where $\Omega\subset\mathbb{R}^{N}$ is a smooth bounded domain, $\lambda\in\mathbb{R}$, $0< s< r< 1 < q < p$, $\gamma>0$ and $j \in \{p, q\}$.

Proposition 3.1

Problem

$$ \textstyle\begin{cases} (-\Delta)_{p}^{r} u=\lambda \vert u \vert ^{p-2}u, & x \in\Omega,\\ u=0, & x\in\mathbb{R}^{N} \setminus\Omega, \end{cases} $$

where $\lambda\in\mathbb{R}$, admits the first eigenvalue $\lambda _{1}(r, p)$given by

$$ \lambda_{1}(r, p):=\inf_{u \in C_{0}^{\infty}(\Omega) \setminus\{0\}} \frac{ \Vert u \Vert _{r,p}^{p}}{\int_{\Omega} \vert u \vert ^{p} \,dx}. $$

Notice that for every $u \in X$ we get $\| u\|_{r,p}^{p} \geq\lambda_{1}(r, p) \int_{\Omega} |u|^{p} \,dx$ and $\lambda_{1}(r, p)$ may be normalized by $\int_{\Omega} |u|^{p} \,dx=1$ as $\lambda_{1}(r, p):=\inf_{u \in C_{0}^{\infty}(\Omega) \setminus\{0\}} \| u\|_{r,p}^{p}$. Also u is the $(r, p)$-eigenfunction of $\lambda _{1}(r,p)$.

Finally, we define the concept of eigenfunction.

Definition 3.1

We say that $u\neq0$, $u \in X$, is a $(r, p)$-eigenfunction of $\lambda_{1}(r,p)$, if for all functions $v \in X$

$$ \begin{gathered} \int_{\mathbb{R}^{2N}}\frac{ ( \vert u(x)-u(y) \vert ^{p-2}(u(x)-u(y)) ) (v(x)-v(y) )}{ \vert x-y \vert ^{N+rp}}\,dx \,dy \\ \quad=\lambda_{1}(r,p) \int_{\Omega}\bigl\vert u(x) \bigr\vert ^{p-2}u(x)v(x) \,dx. \end{gathered} $$

The real number $\lambda_{1}(r,p)$ is called the $(r, p)$-eigenvalue.

Set

$$\lambda_{1}:=\lambda_{1}(i,j)= \textstyle\begin{cases} \lambda_{1}(r,p) & \text{if } i=r, j=p,\\ \lambda_{1}(s,q) & \text{if } i=s, j=q, \end{cases} $$

and

$$ \eta_{1} :=\eta_{1}(j)=\inf _{u \in C_{0}^{\infty}(\Omega) \setminus\{0\}} \textstyle\begin{cases} \frac{\frac{1}{p} \Vert u \Vert _{r,p}^{p}+\frac{\gamma}{q} \Vert u \Vert _{s,q}^{q}}{\frac{1}{p}\int_{\Omega} \vert u \vert ^{p} \,dx} & \text{if } j=p, \\ \frac{\frac{1}{p} \Vert u \Vert _{r,p}^{p}+\frac{\gamma}{q} \Vert u \Vert _{s,q}^{q}}{\frac{1}{q}\int_{\Omega} \vert u \vert ^{q} \,dx} & \text{if } j=q. \end{cases} $$

(3.2)

The number $\eta_{1}$ is called the first generalized eigenvalue of (3.1). Notice that eigenvalues are positive numbers. One can prove the following proposition by almost the same argument as [17, Proposition 1].

Proposition 3.2

$\lambda_{1}=\eta_{1}$and the infimum of $\eta_{1}$in (3.2) is not attained.

Proposition 3.3

Assume that $\lambda< \lambda_{1}$. Then there exist $m_{\lambda}, M_{\lambda}>0$depending on λ such that, for every $u\in X$,

$$ \begin{aligned}[b] m_{\lambda}\biggl( \frac{1}{p} \Vert u \Vert _{r,p}^{p} + \frac{\gamma}{q} \Vert u \Vert _{s,q}^{q} \biggr) &\leq\frac{1}{p} \Vert u \Vert _{r,p}^{p} + \frac{\gamma }{q} \Vert u \Vert _{s,q}^{q}- \frac{\lambda}{p} \int_{\Omega}\bigl\vert u(x) \bigr\vert ^{p} \,dx \\ & \leq M_{\lambda}\biggl(\frac{1}{p} \Vert u \Vert _{r,p}^{p} +\frac{\gamma }{q} \Vert u \Vert _{s,q}^{q} \biggr), \end{aligned} $$

(3.3)

where the constants $m_{\lambda}$and $M_{\lambda}$are $m_{\lambda}:=\min \{ 1, 1-\frac{\lambda}{\lambda_{1}}\}$and $M_{\lambda}:=\max\{ 1, 1-\frac{\lambda}{\lambda_{1}}\}$, respectively.

Proof

The proof is similar to [39, Lemma 10] and it is omitted. □

4 A mountain pass type solution

Here, we study the existence of a mountain pass type solution of (1.2). Notice that problem (1.2) has a variational structure and the functional $I_{\lambda}$ is Fréchet differentiable for $u \in X$ and every $v \in X$,

$$ \begin{aligned} I_{\lambda}' (u)v&= \int_{\mathbb{R}^{2N}}\frac{ ( \vert u(x)-u(y) \vert ^{p-2}(u(x)-u(y)) ) (v(x)-v(y) )}{ \vert x-y \vert ^{N+rp}}\,dx \,dy \\ &\quad+\gamma \int_{\mathbb{R}^{2N}}\frac{ ( \vert u(x)-u(y) \vert ^{q-2}(u(x)-u(y)) ) (v(x)-v(y) )}{ \vert x-y \vert ^{N+sq}}\,dx \,dy \\ &\quad-\lambda \int_{\Omega}\bigl\vert u(x) \bigr\vert ^{p-2}u(x) v(x) \,dx- \int_{\Omega}f\bigl(x, u(x)\bigr)v(x) \,dx. \end{aligned} $$

Hence, weak solutions of (1.2) are critical points of the functional $I_{\lambda}$. Thus one can seek weak solutions as critical points by applying mountain pass theorem (see [2, 32]). In order to apply mountain pass theorem, we show that the functional $I_{\lambda}$ satisfies in the Palais–Smale compactness condition and has a particular geometric structure. For this purpose, we start by proving that the functional $I_{\lambda}$ satisfies the Palais–Smale compactness condition.

Proposition 4.1

Suppose that $\lambda< \lambda_{1}$and f is a Carathéodory function satisfying $(f_{1})$–$(f_{3})$. Let $c \in\mathbb{R}$and $\{u_{n}\}$be a sequence in X such that

$$ I_{\lambda}(u_{n}) \rightarrow c, $$

(4.1)

and

$$ \sup \bigl\{ \bigl|{\prec} I_{\lambda}' (u_{n}), v{\succ}\bigr|: v \in X, \Vert v \Vert _{r, p}=1 \bigr\} \rightarrow0, $$

(4.2)

as $n\rightarrow+\infty$. Then $\{u_{n}\}$is bounded in X.

Proof

Equations (4.1) and (4.2) imply there exists $\sigma>0$ such that for any n

$$ \bigl\vert I_{\lambda}(u_{n}) \bigr\vert \leq\sigma $$

(4.3)

and

$$ \biggl|{\prec} I_{\lambda}' (u_{n}), \frac{u_{n}}{ \Vert u_{n} \Vert _{r, p}}{\succ}\biggr| \leq \sigma. $$

(4.4)

Now, by Lemma 2.2 and $\epsilon=1$ we get

$$ \begin{aligned}[b] & \biggl\vert \int_{\Omega\cap\{ \vert u_{n} \vert \leq\rho\}} \biggl(F\bigl(x, u_{n}(x)\bigr) - \frac{1}{\mu}f\bigl(x, u_{n}(x)\bigr) u_{n}(x) \biggr)\,dx \biggr\vert \\ &\quad\leq\biggl(\rho^{p}+\delta(1)\rho^{b}+ \frac{p}{\mu} \rho^{p} +\frac {b}{\mu}\delta(1) \rho^{b}\biggr) \vert \Omega \vert :=\hat{\sigma}. \end{aligned} $$

(4.5)

Applying Proposition 3.3, then by $(f_{3})$ and (4.5) we have

$$ \begin{aligned}[b] I_{\lambda}(u_{n})- \frac{1}{\mu} {\prec} I_{\lambda}' (u_{n}), u_{n} {\succ}={}& \biggl(\frac{1}{p} - \frac{1}{\mu}\biggr) \bigl( \Vert u_{n} \Vert _{r, p}^{p}- \lambda \Vert u_{n} \Vert _{p}^{p}\bigr)+\gamma\biggl(\frac{1}{q} - \frac{1}{\mu}\biggr) \Vert u_{n} \Vert _{s, q}^{q} \\ &- \frac{1}{\mu} \int_{\Omega}\bigl(\mu F\bigl(x, u_{n}(x)\bigr) -f \bigl(x, u_{n}(x)\bigr) u_{n}(x) \bigr)\,dx \\ \geq& \biggl(\frac{1}{p} -\frac{1}{\mu}\biggr)m_{\lambda} \Vert u_{n} \Vert _{r, p}^{p} \\ &- \int_{\Omega\cap\{ \vert u_{n} \vert \leq\rho\}} \biggl(F\bigl(x, u_{n}(x)\bigr) - \frac {1}{\mu}f\bigl(x, u_{n}(x)\bigr) u_{n}(x) \biggr)\,dx \\ \geq{}& \biggl(\frac{1}{p} -\frac{1}{\mu}\biggr)m_{\lambda} \Vert u_{n} \Vert _{r, p}^{p}-\hat { \sigma}. \end{aligned} $$

(4.6)

From (4.3) and (4.4) one has

$$ I_{\lambda}(u_{n})-\frac{1}{\mu} {\prec} I_{\lambda}' (u_{n}), u_{n} {\succ} \leq\sigma\biggl(1+\frac{1}{\mu} \Vert u_{n} \Vert _{r, p}\biggr). $$

(4.7)

By (4.6) and (4.7) for every $n \in\mathbb{N}$, we have

$$\Vert u_{n} \Vert _{r, p}^{p} \leq \sigma^{*}\biggl(1+\frac{1}{\mu} \Vert u_{n} \Vert _{r, p}\biggr), $$

where $\sigma^{*}$ is a suitable positive constant. Thus the proof is completed. □

Now, we define the operator $A: X \rightarrow X ^{*}$ as follows:

$$ \begin{aligned} {\prec} A(u), v{\succ}:={} & \int_{\mathbb{R}^{2N}}\frac{ ( \vert u(x)-u(y) \vert ^{p-2}(u(x)-u(y)) ) (v(x)-v(y) )}{ \vert x-y \vert ^{N+rp}}\,dx \,dy \\ &+ \gamma \int_{\mathbb{R}^{2N}}\frac{ ( \vert u(x)-u(y) \vert ^{q-2}(u(x)-u(y)) ) (v(x)-v(y) )}{ \vert x-y \vert ^{N+sq}}\,dx \,dy \\ ={}& {\prec} A_{p}(u), v{\succ}+ \gamma{\prec} A_{q}(u), v{\succ} \end{aligned} $$

for every $u,v \in X$. We know that the function $u \in X$ is a weak solution of problem (1.2) such that

$${\prec} A(u), v{\succ}=\lambda \int_{\Omega} \vert u \vert ^{p-2}uv\,dx+ \int_{\Omega}f(x, u)v \,dx, \quad\text{for all } u,v \in X. $$

By [20, 28] it is easy to observe that A verifies the compactness condition as follows:

$(\mathcal{S})$:

If $\{u_{n}\}$ be a sequence in X such that $u_{n} \rightharpoonup u$ (weakly) in X and ${\prec} A(u_{n}), u_{n}-u{\succ}\rightarrow0$ as $n \rightarrow\infty$, then $u_{n} \rightarrow u$ in X as $n \rightarrow\infty$.

Now by property $(\mathcal{S})$, one can prove the next result.

Proposition 4.2

Let f be a Carathéodory function satisfying $(f_{1})$–$(f_{3})$. Let $\{u_{n}\}$be a sequence in X such that $\{u_{n}\}$is bounded in X and (4.2) holds. Then there exists $u \in X$such that, up to a subsequence, $\|u_{n}-u\|_{r, p} \rightarrow0$as $n \rightarrow+ \infty$.

Proof

Since X is a reflexive space and $\{u_{n}\}$ is bounded in X, up to a subsequence if necessary, still denoted by $\{u_{n}\}$, there exists $u \in X$ such that $u_{n} \rightharpoonup u$ in X. By Remark 2.1, up to a subsequence

$$ u_{n} \rightarrow u\quad \text{in } L^{k}\bigl(\mathbb{R}^{N}\bigr),\qquad u_{n} \rightarrow u\quad \text{a.e. in } \mathbb{R}^{N}, $$

(4.8)

as $n \rightarrow+\infty$ and there exists $l \in L^{k}(\mathbb{R}^{N})$ such that

$$ \bigl\vert u_{n}(x) \bigr\vert \leq l(x)\quad \text{a.e. in } \mathbb{R}^{N}, \text{for all } n \in \mathbb{N}, $$

(4.9)

where $k \in[1, p_{r}^{*}[$; for instance, see [9]. By $(f_{1})$, (4.8), (4.9), the fact that the map $t\mapsto f(\cdot, t)$ is continuous in $t \in\mathbb{R}$ and the dominated convergence theorem one can obtain

$$\int_{\Omega}f\bigl(x, u_{n}(x)\bigr) u_{n}(x) \,dx \rightarrow \int_{\Omega}f\bigl(x, u(x)\bigr) u(x) \,dx $$

and

$$\int_{\Omega}f\bigl(x, u_{n}(x)\bigr) u(x) \,dx \rightarrow \int_{\Omega}f\bigl(x, u(x)\bigr) u(x) \,dx, $$

as $n \rightarrow+\infty$. These yield

$$ \lim_{n \rightarrow\infty} \int_{\Omega}f(x, u_{n}) (u_{n}-u) \,dx=0. $$

(4.10)

Now, we recall Simon’s inequality [18]: for any $\zeta, \eta \in\mathbb{R}$ there exists $C_{m}>0$ such that

$$ C_{m}\bigl( \vert \zeta \vert ^{m-2} \zeta- \vert \eta \vert ^{m-2} \eta\bigr). (\zeta- \eta) \geq \textstyle\begin{cases} \vert \zeta- \eta \vert ^{m} & \text{if } m\geq2,\\ \vert \zeta- \eta \vert ^{2} ( \vert \zeta \vert ^{m} + \vert \eta \vert ^{m})^{\frac{m-2}{m}} & \text{if } 1< m< 2. \end{cases} $$

(4.11)

From (4.11), it follows that the operator A is strictly monotone. Suppose $C_{p,q}= \max \{C_{p}, C_{q} \}$. By inequalities (4.11), if $q \geq2$, we get

$$ \begin{aligned}[b] \Vert u_{n} -u \Vert _{r, p}^{p} &\leq \Vert u_{n} -u \Vert _{r, p}^{p}+ \gamma \Vert u_{n} -u \Vert _{s, q}^{q} \\ &\leq C_{p} {\prec} A_{p}u_{n}-A_{p}u, u_{n}-u{\succ}+C_{q} \gamma{\prec} A_{q}u_{n}-A_{q}u, u_{n}-u{\succ} \\ & \leq C_{p,q} {\prec} Au_{n}-Au, u_{n}-u {\succ}. \end{aligned} $$

(4.12)

If $1< q< p<2$, we obtain

$$ \begin{aligned}[b] C_{p,q} {\prec} Au_{n}-Au, u_{n}-u{\succ}&\geq C_{p} {\prec} A_{p}u_{n}-A_{p}u, u_{n}-u{\succ} \\ &\quad+C_{q} \gamma{\prec} A_{q}u_{n}-A_{q}u, u_{n}-u{\succ} \\ & \geq \Vert u_{n} -u \Vert _{r, p}^{2} \bigl( \Vert u_{n} \Vert _{r, p}^{p} + \Vert u \Vert _{r, p}^{p}\bigr)^{\frac {p-2}{p}} \\ & \quad+\gamma \Vert u_{n} -u \Vert _{s, q}^{2} \bigl( \Vert u_{n} \Vert _{s, q}^{q} + \Vert u \Vert _{s, q}^{q}\bigr)^{\frac {q-2}{q}} \\ & \geq \Vert u_{n} -u \Vert _{r, p}^{2} \bigl( \Vert u_{n} \Vert _{r, p}^{p} + \Vert u \Vert _{r, p}^{p}\bigr)^{\frac{p-2}{p}}. \end{aligned} $$

(4.13)

Hence, from (4.13), we get

$$ \begin{aligned}[b] \Vert u_{n} -u \Vert _{r, p}^{p} & \leq C_{p,q}^{\frac{p}{2}} \bigl( {\prec} Au_{n}-Au, u_{n}-u{\succ} \bigr)^{\frac{p}{2}} \bigl( \Vert u_{n} \Vert _{r, p}^{p} + \Vert u \Vert _{r, p}^{p}\bigr)^{\frac{2-p}{2}} \\ & \leq C \bigl({\prec} Au_{n}-Au, u_{n}-u{\succ} \bigr)^{\frac{p}{2}}, \end{aligned} $$

(4.14)

where $C>0$ is a suitable constant. If $1< q<2 \leq p$, we have

$$ \begin{aligned}[b] C_{p,q} {\prec} Au_{n}-Au, u_{n}-u{\succ}&\geq C_{p} {\prec} A_{p}u_{n}-A_{p}u, u_{n}-u{\succ} \\ &\quad+C_{q} \gamma{\prec} A_{q}u_{n}-A_{q}u, u_{n}-u{\succ} \\ & \geq \Vert u_{n} -u \Vert _{r, p}^{p} \\ &\quad +\gamma \Vert u_{n} -u \Vert _{s, q}^{2} \bigl( \Vert u_{n} \Vert _{s, q}^{q} + \Vert u \Vert _{s, q}^{q}\bigr)^{\frac {q-2}{q}} \\ & \geq \Vert u_{n} -u \Vert _{r, p}^{p}. \end{aligned} $$

(4.15)

Moreover, by (4.2), (4.8) and (4.10), we obtain

$$ \begin{aligned} {\prec} A(u_{n}), u_{n}-u {\succ}&={\prec} I_{\lambda}'(u_{n}), u_{n}-u{\succ} \\ &\quad+\lambda \Vert u_{n}-u \Vert _{p}^{p}+ \int_{\Omega}f(x, u_{n}-u) (u_{n}-u) \,dx \rightarrow0. \end{aligned} $$

Combining (4.12), (4.14) and (4.15), by property $(\mathcal{S})$, one can conclude that $u_{n} \rightarrow u$ in X as $n \rightarrow\infty$. □

Now, we investigate that the functional $I_{\lambda}$ satisfies in the geometry of mountain pass theorem. Here, we prove two next propositions in the same ways of [39, Propositions 11 and 12].

Proposition 4.3

Assume that $\lambda< \lambda_{1}$and f is a Carathéodory function satisfying $(f_{1})$and $(f_{2})$. Then there exist $\rho>0$and $\alpha>0$such that for any $u \in X$with $\|u\|_{r, p}=\rho$it implies that $I_{\lambda}(u)\geq\alpha$.

Proof

Let u be a function in X. From (2.2) and (3.3) for every $\epsilon>0$, we obtain

$$ \begin{aligned}[b] I_{\lambda}(u)&\geq \frac{1}{p} \Vert u \Vert _{r, p}^{p}+ \frac{\gamma}{q} \Vert u \Vert _{s, q}^{q}- \frac{\lambda}{p} \int_{\Omega}\bigl\vert u(x) \bigr\vert ^{p} \,dx- \epsilon \int _{\Omega}\bigl\vert u(x) \bigr\vert ^{p} \,dx-\delta( \epsilon) \int_{\Omega}\bigl\vert u(x) \bigr\vert ^{b} \,dx\hspace{-12pt} \\ & \geq\frac{m_{\lambda}}{p} \Vert u \Vert _{r, p}^{p}- \epsilon \vert \Omega \vert ^{\frac {p_{r}^{*}-p}{p_{r}^{*}}} \Vert u \Vert _{p_{r}^{*}}^{p} -\delta( \epsilon) \vert \Omega \vert ^{\frac {p_{r}^{*}-b}{p_{r}^{*}}} \Vert u \Vert _{p_{r}^{*}}^{b}. \end{aligned} $$

(4.16)

By the fact that $L^{p_{r}^{*}}(\Omega) \hookrightarrow L^{p}(\Omega)$ and $L^{p_{r}^{*}}(\Omega) \hookrightarrow L^{b}(\Omega)$ are continuous ($\max \{p, b\}=b< p_{r}^{*}$) and thanks to Remark 2.1, from (4.16) for every $\epsilon>0$, we get

$$ \begin{aligned}[b] I_{\lambda}(u) \geq& \frac{m_{\lambda}}{p} \Vert u \Vert _{r, p}^{p} - \epsilon c \vert \Omega \vert ^{\frac{p_{r}^{*}-p}{p_{r}^{*}}} \Vert u \Vert _{r, p}^{p}-\delta( \epsilon) c^{\frac{b}{p}} \vert \Omega \vert ^{\frac{p_{r}^{*}-b}{p_{r}^{*}}} \Vert u \Vert _{r, p}^{b} \\ = & \biggl( \frac{m_{\lambda}}{p}-\epsilon c \vert \Omega \vert ^{\frac {p_{r}^{*}-p}{p_{r}^{*}}} \biggr) \Vert u \Vert _{r, p}^{p} - \delta( \epsilon) c^{\frac {b}{p}} \vert \Omega \vert ^{\frac{p_{r}^{*}-b}{p_{r}^{*}}} \Vert u \Vert _{r, p}^{b}. \end{aligned} $$

(4.17)

Choosing $\epsilon>0$ such that $m_{\lambda}-p \epsilon c |\Omega |^{\frac{p_{r}^{*}-p}{p_{r}^{*}}}>0$, from (4.17) one has

$$ I_{\lambda}(u) \geq\beta \Vert u \Vert _{r, p}^{p} \bigl(1- \theta \Vert u \Vert _{r, p}^{b-p} \bigr), $$

for some positive constants β and θ. Let u in X such that $\|u\|_{r, p}=\rho>0$. Since $b>p$, by choosing ρ small enough, that is, choosing ρ such that $1-\theta\rho^{b-p}>0$, we get

$$ \inf_{u \in X, \|u\|_{r, p}=\rho} I_{\lambda}(u) \geq\beta\rho ^{p}\bigl(1-\theta\rho^{b-p}\bigr):=\alpha>0. $$

Thus the proof is completed. □

Proposition 4.4

Suppose $\lambda< \lambda_{1}$and f is a Carathéodory function satisfying $(f_{1})$–$(f_{3})$. Then there exists $e \in X$such that $e \geq0$a.e. in $\mathbb{R}^{N}$with $\|e\|_{r, p}>\rho$and $I_{\lambda}(e)< \alpha$, where ρ and α are given in Proposition 4.3.

Proof

Let $u \in X$ be fixed such that $\|u\|_{r, p}=1$ and $u\geq0$ a.e. in $\mathbb{R}^{N}$. Also, suppose $t>0$, from Lemma 2.3 and Proposition 3.3, we get

$$ \begin{aligned} I_{\lambda}(tu) &= \frac{1}{p} \int_{\mathbb{R}^{2N}} \frac { \vert tu(x)-tu(y) \vert ^{p}}{ \vert x-y \vert ^{N+rp}}\,dx \,dy + \frac{\gamma}{q} \int _{\mathbb{R}^{2N}} \frac{ \vert tu(x)-tu(y) \vert ^{q}}{ \vert x-y \vert ^{N+sq}}\,dx \,dy \\ &\quad-\frac{\lambda}{p} \int_{\Omega}\bigl\vert tu(x) \bigr\vert ^{p} \,dx- \int_{\Omega}F\bigl(x, tu(x)\bigr) \,dx \\ & \leq\frac{M_{\lambda}}{p}t^{p}+\frac{M_{\lambda}}{q}\gamma t^{q} \Vert u \Vert _{s, q}^{q} -t^{\mu}\int_{\Omega}m(x) \bigl\vert u(x) \bigr\vert ^{\mu}\,dx+ \int_{\Omega}M(x) \,dx. \end{aligned} $$

Since $\mu>p$, as $t \rightarrow+\infty$, we get $I_{\lambda}(tu)\rightarrow-\infty$. Thus we can conclude the assertion by $e=\tau u$ such that τ is enough large. □

Propositions 4.3 and 4.4 imply the geometry of the mountain pass theorem. Thus we are ready to present the proof of Theorem 1.1.

Proof of Theorem 1.1

Since Propositions 4.1–4.4 hold, the mountain pass theorem admits that there exists a critical point $u \in X$ of $I_{\lambda}$. Moreover,

$$I_{\lambda}(u) \geq\alpha>0=I_{\lambda}(0), $$

so $u\neq0$. Then the existence of one weak solution $u \in X$ (i.e. a mountain pass type solution) is proved. □