1 Introduction

The dynamic relationship between predator and prey has long been and will continue to be one of the dominant themes in both ecology and mathematical ecology due to its universal existence and importance [1]. Furthermore, the study of dynamic behaviors of predator–prey system incorporating a prey refuge become one of the most important research topic; see [231]. In [2], Amant proposed a Lotka–Volterra predator–prey model with a constant prey refuge:

$$ \begin{gathered} \dot{x}(t)=rx-\beta (x-m)y, \\ \dot{y}(t)=e(x-m)y-dy, \end{gathered} $$
(1.1)

where \(x(t)\) and \(y(t)\) denote the densities of prey and predator populations at time t, respectively, r is the growth rate of the prey, β is the per capita rate of predation, m is the constant prey refuge, e is a conversion rate of eaten prey into new predator abundance, and d is the per capita death rate of the predator.

Considering there is an upper limit on how much predators can eat, the response of the predator to the bait should be a function of the saturation factor. In [3], Gonzlez-Olivares and Ramos-Jiliberto investigated the dynamic behaviors of predator–prey system incorporating Holling type II and a constant prey refuge:

$$ \begin{gathered} \dot{x}(t)=rx \biggl(1- \frac{x}{K} \biggr)- \frac{\beta (x-m)y}{(x-m)+a}, \\ \dot{y}(t)=-dy+ \frac{c\beta (x-m)y}{(x-m)+a}, \end{gathered} $$
(1.2)

where K is the prey environmental carrying capacity, a is the amount of prey needed to achieve one-half of β. They show that the effect of prey refuges would have a stabilizing influence on the dynamical consequences of system (1.2).

Some scholars argued that the nonautonomous case is more realistic, because many biological or environmental parameters do subject to fluctuate with time, thus more complex equations should be introduced. Many scholars studied the dynamic behaviors of nonautonomous predator–prey system incorporating prey refuge. In [4], Zhu proposed and studied the nonautonomous predator–prey system incorporating prey refuge:

$$ \begin{gathered} \dot{x}(t)=x(t)\bigl[a(t)-b(t)x(t)\bigr]-c(t) \bigl(x(t)-m(t)\bigr)y(t), \\ \dot{y}(t)=y(t)\bigl[-r_{1}(t)-k(t)y(t)+d(t) \bigl(x(t)-m(t)\bigr) \bigr], \end{gathered} $$
(1.3)

where \(x(t)\) and \(y(t)\) denote the density of prey and predator populations at time t, respectively. \(a(t)\), \(b(t)\), \(c(t)\), \(m(t)\), \(r_{1}(t)\), \(k(t)\), \(d(t)\) are nonnegative continuous function that have the upper and lower bounds. \(a(t)\) is the intrinsic per capita growth rate of prey, \(b(t)\) is the per capita death rate of prey, \(c(t)\) is the maximal per capita consumption rate of predators, \(m(t)\) is the maximum capacity of refuge, \(r_{1}(t)\) is the per capita death rate of predator, \(k(t)\) is the density constraints on predator populations, and \(d(t)/c(t)\) is the conversion coefficient. In [4], the sufficient conditions to guarantee the global asymptotic stability of the system (1.3) are obtained. On the basis of system (1.3), Wu [5] further studied the extinction of predator populations. In this paper, we study the nonautonomous predator–prey system incorporating prey refuge and Holling type II schemes:

$$ \begin{gathered} \dot{x}(t)=x(t)\bigl[a(t)-b(t)x(t)\bigr]- \frac{c(t)(x(t)-m(t))y(t)}{a_{1}(t)+x(t)-m(t)}, \\ \dot{y}(t)=y(t) \biggl[-r_{1}(t)-k(t)y(t)+ \frac{d(t)(x(t)-m(t))}{a_{1}(t)+x(t)-m(t)} \biggr], \end{gathered} $$
(1.4)

where \(a_{1}(t)\) is the amount of prey needed to achieve one-half of \(c(t)\).

Based on the biological significance of systems, we consider system (1.4) together with the following initial conditions:

$$ x(0)>0,\qquad y(0)>0. $$
(1.5)

Furthermore, for a bounded continuous function \(g(t)\) defined on R,

$$ g^{l}=\inf_{t\in R}g(t), \qquad g^{u}=\sup _{t\in R}g(t). $$

The following work is organized as follows. Sufficient conditions which guarantee the positive and permanence of system (1.4) are given in Sect. 2. In Sect. 3, we obtained a set of sufficient conditions for global stability of the system (1.4). In Sect. 4, the extinction of predator are studied and a set of sufficient conditions that guarantee the extinction of predator are obtained. In Sect. 5, three examples together with their numerical simulations show the feasibility of the main results. This paper ends by a brief conclusion.

2 Positive and permanence

Lemma 2.1

If the system (1.4) satisfies

$$ a_{1}^{l}>m^{u}, $$
(2.1)

then \(R_{+}^{2}=\{(x,y)|x>0,y>0\}\) is a invariant set of the system (1.4).

Proof

Let \((x(t),y(t))^{T}\) be any positive solution of the system (1.4) that satisfies the initial condition (1.5). From the first equation of the system (1.4), it follows that

$$\begin{aligned}& \dot{x}|_{x=0} = \frac{c(t)m(t)y(t)}{a_{1}(t)-m(t)}, \\& y(t)=y(0)\exp \biggl\{ \int _{0}^{t} \biggl[-r_{1}(s)-k(s)y(s)+ \frac{d(s)(x(s)-m(s))}{a_{1}(s)+x(s)-m(s)} \biggr]\,ds \biggr\} . \end{aligned}$$

Therefore, Lemma 2.1 is true when the conditon (2.1) is true and \(x(0)>0\), \(y(0)>0\). □

Lemma 2.2

For every positive solution \((x(t),y(t))^{T}\) that satisfies the initial condition (1.5), if the system satisfies (2.1) and

$$ a^{l}> \frac{c^{u}M_{2}}{a_{1}^{l}-m^{u}};\qquad r_{1}^{u}< \frac{d^{l}(m_{1}-m^{u})}{a_{1}^{u}+M_{1}};\qquad m_{1}>m^{u}. $$
(2.2)

then, for every positive solution \((x(t),y(t))^{T}\) that satisfies the initial condition (1.5), system (1.4) is permanent. That is,

$$ \limsup_{t\rightarrow \infty }x(t)\leq M_{1},\qquad \limsup _{t \rightarrow \infty }y(t)\leq M_{2},\qquad \liminf _{t\rightarrow \infty }x(t) \geq m_{1},\qquad \liminf_{t\rightarrow \infty }y(t) \geq m_{2}. $$

Here,

$$\begin{aligned}& M_{1}= \frac{a^{u}}{b^{l}},\qquad M_{2}= \frac{d^{u}M_{1}}{k^{l}(a_{1}^{l}-m^{u})}, \\& m_{1}= \frac{a^{l}(a_{1}^{l}-m^{u})-c^{u}M_{2}}{b^{u}(a_{1}^{l}-m^{u})},\qquad m_{2}= \frac{d^{l}(m_{1}-m^{u})-r_{1}^{u}(a_{1}^{u}+M_{1})}{k^{u}(a_{1}^{u}+M_{1})}. \end{aligned}$$

Proof

For every positive solution \((x(t),y(t))^{T}\) that satisfies the initial condition (1.5), from the first equation of system (1.4), condition (2.1) and the third condition of (2.2), it follows that

$$ \begin{aligned} \dot{x}(t)&=x(t)\bigl[a(t)-b(t)x(t)\bigr]- \frac{c(t)(x(t)-m(t))y(t)}{a_{1}(t)+x(t)-m(t)} \\ &\leq x(t)\bigl[a^{u}-b^{l}x(t)\bigr]. \end{aligned} $$
(2.3)

Hence, according to Lemma 2.3 in [32], it is directly found that

$$ \limsup_{t\rightarrow \infty }x(t)\leq \frac{a^{u}}{b^{l}}{:=}M_{1}. $$
(2.4)

For any small positive constant \(\varepsilon >0\), there exists \(T_{1}>0\) such that, for all \(t \geq T_{1}\),

$$ x(t)< M_{1}+\varepsilon . $$
(2.5)

It follows from (2.5) and the second equation of the system (1.4) that

$$ \begin{aligned} \dot{y}(t)&=y(t) \biggl[-r_{1}(t)-k(t)y(t)+ \frac{d(t)(x(t)-m(t))}{a_{1}(t)+x(t)-m(t)} \biggr] \\ &\leq y(t) \biggl[-k(t)y(t)+ \frac{d(t)x(t)}{a_{1}(t)-m(t)} \biggr] \\ &\leq y(t) \biggl[-k^{l}y(t)+ \frac{d^{u}(M_{1}+\varepsilon )}{a_{1}^{l}-m^{u}} \biggr]. \end{aligned} $$
(2.6)

According to Lemma 2.3 in [32], it follows that

$$ \limsup_{t\rightarrow \infty }y(t)\leq \frac{d^{u}(M_{1}+\varepsilon )}{k^{l}(a_{1}^{l}-m^{u})}. $$
(2.7)

Because of the arbitrariness of ε, let \(\varepsilon \rightarrow 0\), it follows from (2.7) that

$$ \limsup_{t\rightarrow \infty }y(t)\leq \frac{d^{u}M_{1}}{k^{l}(a_{1}^{l}-m^{u})}{:=}M_{2}. $$
(2.8)

Conditions (2.8) implies that, for any small \(\varepsilon >0\), there exists a \(T_{2}>T_{1}\), such that, for all \(t>T_{2}\),

$$ y(t)< M_{2}+\varepsilon . $$
(2.9)

Then, for \(t>T_{2}\), from the first equation of system (1.4), it follows that

$$ \begin{aligned} \dot{x}(t)&=x(t)\bigl[a(t)-b(t)x(t)\bigr]- \frac{c(t)(x(t)-m(t))y(t)}{a_{1}(t)+x(t)-m(t)} \\ &\geq x(t) \biggl[a(t)-b(t)x(t)- \frac{c(t)y(t)}{a_{1}(t)-m(t)} \biggr] \\ &\geq x(t) \biggl[a^{l}-b^{u}x(t)- \frac{c^{u}(M_{2}+\varepsilon )}{a_{1}^{l}-m^{u}} \biggr]. \end{aligned} $$
(2.10)

According to Lemma 2.3 in [32], it follows that

$$ \liminf_{t\rightarrow \infty }x(t)\geq \frac{a^{l}- \frac{c^{u}(M_{2}+\varepsilon )}{a_{1}^{l}-m^{u}}}{b^{u}}. $$
(2.11)

Let \(\varepsilon \rightarrow 0\), then

$$ \liminf_{t\rightarrow \infty }x(t)\geq \frac{a^{l}(a_{1}^{l}-m^{u})-c^{u}M_{2}}{b^{u}(a_{1}^{l}-m^{u})}{:=}m_{1}. $$
(2.12)

Let \(0<\varepsilon <\frac{1}{2}m_{1}\) be any positive constant small enough. Then it follows from (2.12) that there exists a \(T_{3}>T_{2}\), such that, for all \(t>T_{3}\),

$$ x(t)>m_{1}-\varepsilon . $$
(2.13)

From the second equation of the system (1.4) together with (2.5) and (2.13), it follows

$$ \begin{aligned} \dot{y}(t)&=y(t) \biggl[-r_{1}(t)-k(t)y(t)+ \frac{d(t)(x(t)-m(t))}{a_{1}(t)+x(t)-m(t)} \biggr] \\ &\geq y(t) \biggl[-r_{1}(t)-k(t)y(t)+ \frac{d(t)(x(t)-m(t))}{a_{1}(t)+x(t)} \biggr] \\ &\geq y(t) \biggl[-r_{1}^{u}-k^{u}y(t)+ \frac{d^{l}(m_{1}-\varepsilon -m^{u})}{a_{1}^{u}+M_{1}+\varepsilon } \biggr]. \end{aligned} $$
(2.14)

According to Lemma 2.3 in [32], it follows that

$$ \liminf_{t\rightarrow \infty }y(t)\geq \frac{ \frac{d^{l}(m_{1}-\varepsilon -m^{u})}{a_{1}^{u}+M_{1}+\varepsilon }-r_{1}^{u}}{k^{u}}. $$
(2.15)

Let \(\varepsilon \rightarrow 0\), then

$$ \liminf_{t\rightarrow \infty }y(t)\geq \frac{ \frac{d^{l}(m_{1}-m^{u})}{a_{1}^{u}+M_{1}}-r_{1}^{u}}{k^{u}}{:=}m_{2}. $$
(2.16)

This completes the proof of Lemma 2.2. □

3 Global stability

We introduce some notations before we state the main result of this section. Set

$$ \begin{gathered} \delta _{1}(t) = a_{1}(t)+x(t)-m(t), \\ \delta _{2}(t) = a_{1}(t)+x_{1}(t)-m(t), \\ \Delta (m_{1}) = a_{1}^{l}+(m_{1}- \varepsilon )-m^{u}, \\ A = b^{l}- \frac{c^{u}M_{2}+d^{u}a_{1}^{u}}{[\Delta (m_{1})]^{2}}, \\ B = k^{l}- \frac{c^{u}m^{u}}{\Delta (m_{1})m_{1}}. \end{gathered} $$
(3.1)

Theorem 3.1

Let \((x(t), y(t))^{T}\), \((x_{1}(t), y_{1}(t))^{T}\) be the positive solutions of system (1.4), assume system (1.4) satisfies all the conditions of Lemma 2.2, assume further that

$$ A< 0,\qquad B< 0, $$
(3.2)

then

$$ \lim_{t\rightarrow +\infty } \bigl[ \bigl\vert x(t)-x_{1}(t) \bigr\vert + \bigl\vert y(t)-y_{1}(t) \bigr\vert \bigr]=0. $$
(3.3)

That is, the system (1.4) shows global stability.

Proof

From (3.1), it follows that

$$ \Delta \bigl(m_{1}^{\varepsilon }\bigr)=a_{1}^{l}+(m_{1}- \varepsilon )-m^{u}. $$
(3.4)

Let

$$ \begin{gathered} V_{1}(t)= \bigl\vert \ln x(t)-\ln x_{1}(t) \bigr\vert , \\ V_{2}(t)= \bigl\vert \ln y(t)-\ln y_{1}(t) \bigr\vert . \end{gathered} $$
(3.5)

Then, for \(t>T\), we have

$$ \begin{aligned} D^{+}V_{1}(t)&=sgn \bigl(x(t)-x_{1}(t)\bigr) \biggl(-b(t) \bigl(x(t)-x_{1}(t) \bigr)+ \frac{c(t)y_{1}(t)}{\delta _{2}(t)}- \frac{c(t)y(t)}{\delta _{1}(t)} \\ &\quad {}+ \frac{c(t)m(t)y(t)}{\delta _{1}(t)x(t)}- \frac{c(t)m(t)y_{1}(t)}{\delta _{2}(t)x_{1}(t)} \biggr) \\ &=sgn \bigl(x(t)-x_{1}(t)\bigr) (-b(t) \bigl(x(t)-x_{1}(t) \bigr)-\bigl(y(t)-y_{1}(t)\bigr) \frac{c(t)}{\delta _{1}(t)} \\ &\quad {}+\bigl(x(t)-x_{1}(t)\bigr) \frac{c(t)y(t)}{\delta _{1}(t)\delta _{2}(t)}+ \frac{c(t)m(t)}{\delta _{1}(t)x(t)}\bigl(y(t)-y_{1}(t)\bigr) \\ &\quad {}- \frac{c(t)m(t)y_{1}(t)}{\delta _{1}(t)x(t)x_{1}(t)}\biggl(x(t)-x_{1}(t)- \frac{c(t)m(t)y_{1}(t)}{\delta _{1}(t)\delta _{2}(t)x_{1}(t)} \bigl(x(t)-x_{1}(t)\bigr)\biggr) \\ &= \biggl(-b(t)+ \frac{c(t)y(t)}{\delta _{1}(t)\delta _{2}(t)}- \frac{c(t)m(t)y_{1}(t)}{\delta _{1}(t)x(t)x_{1}(t)} - \frac{c(t)m(t)y_{1}(t)}{\delta _{1}(t)\delta _{2}(t)x_{1}(t)} \biggr) \bigl\vert x(t)-x_{1}(t) \bigr\vert \\ &\quad {}+ \biggl( \frac{c(t)m(t)}{\delta _{1}(t)x(t)}- \frac{c(t)}{\delta _{1}(t)} \biggr) \bigl\vert y(t)-y_{1}(t) \bigr\vert \\ &\leq \biggl(-b(t)+ \frac{c(t)y(t)}{\delta _{1}(t)\delta _{2}(t)} \biggr) \bigl\vert x(t)-x_{1}(t) \bigr\vert + \frac{c(t)m(t)}{\delta _{1}(t)x(t)} \bigl\vert y(t)-y_{1}(t) \bigr\vert \\ &\leq \biggl(-b^{l}+ \frac{c^{u}(M_{2}+\varepsilon )}{[\Delta (m_{1}^{\varepsilon })]^{2}} \biggr) \bigl\vert x(t)-x_{1}(t) \bigr\vert + \frac{c^{u}m^{u}}{\Delta (m_{1}^{\varepsilon })(m_{1}-\varepsilon )} \bigl\vert y(t)-y_{1}(t) \bigr\vert \end{aligned} $$

and

$$ \begin{aligned} &D^{+}V_{2}(t)\\ &\quad =sgn \bigl(y(t)-y_{1}(t)\bigr) \biggl(-k(t) \bigl(y(t)-y_{1}(t) \bigr)+ \frac{d(t)(x(t)-m(t))}{\delta _{1}(t)}- \frac{d(t)(x_{1}(t)-m(t))}{\delta _{2}(t)} \biggr) \\ &\quad =sgn\bigl(y(t)-y_{1}(t)\bigr) \biggl(-k(t) \bigl(y(t)-y_{1}(t) \bigr)+ \frac{d(t)(x(t)-m(t))}{\delta _{1}(t)}- \frac{d(t)(x_{1}(t)-m(t))}{\delta _{2}(t)} \biggr) \\ &\quad =-k(t) \bigl\vert y(t)-y_{1}(t) \bigr\vert + \frac{d(t)a_{1}(t)}{\delta _{1}(t)\delta _{2}(t)} \bigl\vert x(t)-x_{1}(t) \bigr\vert \\ &\quad \leq -k^{l} \bigl\vert y(t)-y_{1}(t) \bigr\vert + \frac{d^{u}a_{1}^{u}}{[\Delta (m_{1}^{\varepsilon })]^{2}} \bigl\vert x(t)-x_{1}(t) \bigr\vert . \end{aligned} $$

Set

$$ V(t)=V_{1}(t)+V_{2}(t). $$
(3.6)

Then

$$ \begin{aligned} D^{+}V(t)&\leq \biggl(-b^{l}+ \frac{c^{u}(M_{2}+\varepsilon )+d^{u}a_{1}^{u}}{[\Delta (m_{1}^{\varepsilon })]^{2}} \biggr) \bigl\vert x(t)-x_{1}(t) \bigr\vert \\ &\quad {} + \biggl(-k^{l}+ \frac{c^{u}m^{u}}{\Delta (m_{1}^{\varepsilon })(m_{1}-\varepsilon )} \biggr) \bigl\vert y(t)-y_{1}(t) \bigr\vert . \end{aligned} $$
(3.7)

Let \(\varepsilon \rightarrow 0\), then

$$ \begin{aligned} D^{+}V(t)&\leq \biggl(-b^{l}+ \frac{c^{u}M_{2}+d^{u}a_{1}^{u}}{[\Delta (m_{1})]^{2}} \biggr) \bigl\vert x(t)-x_{1}(t) \bigr\vert + \biggl(-k^{l}+ \frac{c^{u}m^{u}}{\Delta (m_{1})m_{1}} \biggr) \bigl\vert y(t)-y_{1}(t) \bigr\vert \\ &=-A \bigl\vert x(t)-x_{1}(t) \bigr\vert -B \bigl\vert y(t)-y_{1}(t) \bigr\vert . \end{aligned} $$
(3.8)

Here A and B are defined in Eq. (3.1). Setting \(\alpha =\min \{A,B\}>0\). Then \((x(t),y(t))\) is stable under the meaning of Lyapunov. Integrating Eq. (3.8) from T to t, then, for \(t>T\), we get

$$ V(t)-V(T)\leq -\alpha \int _{T}^{t}\bigl[ \bigl\vert x(s)-x_{1}(s) \bigr\vert + \bigl\vert y(s)-y_{1}(s) \bigr\vert \bigr] \, \mathrm{d}s, $$
(3.9)

hence,

$$ V(t)+\alpha \int _{T}^{t}\bigl[ \bigl\vert x(s)-x_{1}(s) \bigr\vert + \bigl\vert y(s)-y_{1}(s) \bigr\vert \bigr] \, \mathrm{d}s\leq V(T)< +\infty . $$
(3.10)

Therefore, \(V(t)\) is bounded on the interval \([T,\infty ]\) and

$$ \int _{T}^{t} \bigl\vert x(s)-x_{1}(s) \bigr\vert \,\mathrm{d}s< +\infty , \int _{T}^{t} \bigl\vert y(s)-y_{1}(s) \bigr\vert \,\mathrm{d}s< +\infty . $$
(3.11)

So \(|x(t)-x_{1}(t)|\) and \(|y(t)-y_{1}(t)|\) are integrable on the interval \([T, +\infty ]\). On the other hand, it is easy to see that \(\dot{x}(t)\), \(\dot{y}(t)\), \(\dot{x_{1}}(t)\), \(\dot{y_{1}}(t)\) are bounded. Therefore, \(|x(t)-x_{1}(t)|\) and \(|y(t)-y_{1}(t)|\) are uniformly continuous on interval \([T, +\infty ]\). By the Barbalat lemma, it gives

$$ \lim_{t\rightarrow +\infty } \bigl[ \bigl\vert x(t)-x_{1}(t) \bigr\vert + \bigl\vert y(t)-y_{1}(t) \bigr\vert \bigr]=0. $$
(3.12)

Then system (1.4) shows global stability. This completes the proof of Theorem 3.1. □

4 The extinction of predator

Consider the following equation:

$$ \dot{x}(t)=x(t)\bigl[a(t)-b(t)x(t)\bigr], $$
(4.1)

where \(a(t)\) and \(b(t)\) are continuous functions defined in the real domain R and are bounded above and below by positive constants. From Lemma 4.1 of [33], Eq. (4.1) has a solution \(x_{*}(t)\) bounded above and below by positive constants and

$$ x_{*}(t)= \biggl[ \int _{-\infty }^{t} d(s)e^{-\int _{s}^{t} a(\tau ) \,\mathrm{d}\tau } \,\mathrm{d}s \biggr]^{-1}. $$
(4.2)

Theorem 4.1

If the system (1.4) satisfies condition (2.1) and

$$ -r_{1}^{l}+ \frac{b^{u} (\max \{ \frac{a^{u}}{b^{l}},m^{u} \}-m^{l} )}{a_{1}^{l}-m^{u}}< 0, $$
(4.3)

then the predator population will go extinct, that is the solution of the system (1.4) satisfies

$$ \lim_{t\rightarrow \infty }y(t)=0. $$

Proof

Let \((x(t),y(t))\) satisfying condition (1.5) be any positive solution of system (1.4). It is discussed in three cases.

(1) Assume that \(x(t)>m(t)\) is true for all \(t>0\). Let \(\tilde{x}(t)\) be the solution of (4.1) with initial \(\tilde{x}(0)=x(0)\). It follows from Lemma 4.1 in [33] that

$$ \lim_{t\rightarrow \infty }\bigl(\tilde{x}(t)-x_{*}(t)\bigr)=0. $$
(4.4)

Then, for any arbitrary \(\varepsilon >0\), there exists a \(T_{1} (>T)\), such that

$$ \bigl\vert \tilde{x}(t)-x_{*}(t) \bigr\vert < \varepsilon ,\quad t \geq T_{1}. $$
(4.5)

From the first equation of system (1.4), it follows that

$$ \dot{x}(t)=x(t)\bigl[a(t)-b(t)x(t)\bigr]- \frac{c(t)(x(t)-m(t))}{a_{1}(t)+x(t)-m(t)}\leq x(t) \bigl[r(t)-d(t)x(t)\bigr]. $$

By the applying comparison theorem of differential equations, we get

$$ x(t)\leq \tilde{x}(t),\quad t\geq 0. $$

Thus, combined with (4.5), we have

$$ x(t)\leq \tilde{x}(t)=x_{*}(t)+\bigl(\tilde{x}(t)-x_{*}(t) \bigr)< x_{*}(t)+ \varepsilon ,\quad t\geq T_{1}. $$

Therefore, from the second equation of system (1.4), it follows that, for \(t\geq T_{1}\),

$$ \begin{aligned} \dot{y}(t)&=y(t) \biggl[-r_{1}(t)-k(t)y(t)+ \frac{d(t)(x(t)-m(t))}{a_{1}(t)+x(t)-m(t)} \biggr] \\ &\leq y(t) \biggl[-r_{1}(t)+ \frac{d(t)(x_{*}(t)+\varepsilon -m(t))}{a_{1}(t)-m(t)} \biggr]. \end{aligned} $$
(4.6)

Integrating Eq. (4.6) from \(T_{1}\) to t, we get

$$ \begin{aligned} y(t)&\leq y(T_{1})e^{\int _{T_{1}}^{t} (-r_{1}(s)+ \frac{d(s)(x_{*}(s)+\varepsilon -m(s))}{a_{1}(s)-m(s)} ) \,\mathrm{d}s} \\ &\leq y(T_{1})e^{\int _{T_{1}}^{t} (-r_{1}(s)+ \frac{d(s) ( \frac{a^{u}}{b^{l}}+\varepsilon -m(s) )}{a_{1}(s)-m(s)} ) \,\mathrm{d}s} \\ &\leq y(T_{1})e^{\int _{T_{1}}^{t} (-r_{1}^{l}+ \frac{d^{u} ( \frac{a^{u}}{b^{l}}+\varepsilon -m^{l} )}{a_{1}^{l}-m^{u}} ) \,\mathrm{d}s}. \end{aligned} $$
(4.7)

Let \(\varepsilon \rightarrow 0\), we have

$$ y(t)\leq y(T_{1})e^{\int _{T_{1}}^{t} (-r_{1}^{l}+ \frac{d^{u} ( \frac{a^{u}}{b^{l}}-m^{l} )}{a_{1}^{l}-m^{u}} ) \,\mathrm{d}s}. $$

Therefore, from the condition (4.3), it follows that

$$ \lim_{t\rightarrow \infty }y(t)=0. $$

(2) Assume that \(x(t)< m(t)\) is true for all \(t>0\). Then, from the second equation of system (1.4) and Lemma 2.1, it follows that

$$ \begin{aligned} \dot{y}(t)&=y(t) \biggl[-r_{1}(t)-k(t)y(t)+ \frac{d(t)(x(t)-m(t))}{a_{1}(t)+x(t)-m(t)} \biggr] \\ &\leq -k(t)y(t). \end{aligned} $$
(4.8)

Integrating Eq. (4.8) from 0 to t, we get

$$ y(t)\leq y(0)e^{\int _{0}^{t}(-k(s)) \,\mathrm{d}s}\leq y(0)e^{-k^{l}t}. $$

Therefore,

$$ \lim_{t\rightarrow \infty }y(t)=0. $$

(3) Assume that \(x(t)\) oscillates with respect to \(m(t)\). In this case, suppose \(x(t)\) and \(m(t)\) intersect each other at the point \(t_{i}\), \(i=0,1,2,\ldots \) , and

$$ x(t)>m(t), \qquad t\in (t_{2k+1}, t_{2k+2});\qquad x(t)< m(t),\qquad t\in (t_{2k+2}, t_{2k+3}). $$

Now assume that \(x(t)\) maximizes at the point \(\tau _{k}\in (t_{2k+1},t_{2k+2})\), \(k=0,1,2,\ldots \) , then \(x(\tau _{k})>m(\tau _{k})\) and \(\dot{x}(t)|_{t=\tau _{k}}=0\).

From the first equation of system (1.4), it follows that

$$ \dot{x}(t)|_{t=\tau _{k}}=x(\tau _{k})\bigl[a(\tau _{k})-b(\tau _{k})x( \tau _{k})\bigr]- \frac{c(\tau _{k})(x(\tau _{k})-m(\tau _{k}))y(\tau _{k})}{a_{1}(\tau _{k})+x(\tau _{k})-m(\tau _{k})}=0. $$

Then

$$ x(\tau _{k})\bigl[a(\tau _{k})-b(\tau _{k})x( \tau _{k})\bigr]>0. $$
(4.9)

By Lemma 2.1, note that \(x(\tau _{k})>0\), and combining with Eq. (4.9), \(a(\tau _{k})-b(\tau _{k})x(\tau _{k})>0\), that is,

$$ x(\tau _{k})< \frac{a(\tau _{k})}{b(\tau _{k})}\leq \frac{a^{u}}{b^{l}}. $$

Since \(x(t)\) is maximized at the point \(\tau _{k}\), the above analysis shows that \(x(t)<\frac{a^{u}}{b^{l}}\), \(t\in (t_{2k+1},t_{2k+2})\), \(k=0,1,2, \ldots \)Ȧt the same time, \(x(t)< m(t)< m^{u}\), \(t\in (t_{2k+2},t_{2k+3})\). Then, for all \(t\geq t_{1}\),

$$ x(t)< \max \biggl\{ \frac{a^{u}}{b^{l}},m^{u} \biggr\} . $$
(4.10)

Substituting Eq. (4.10) into the second formula of the system (1.4), we get, for all \(t\geq t_{1}\),

$$ \begin{aligned} \dot{y}(t)&=y(t) \biggl[-r_{1}(t)-k(t)y(t)+ \frac{d(t)(x(t)-m(t))}{a_{1}(t)+x(t)-m(t)} \biggr] \\ &\leq y(t) \biggl[-r_{1}(t)+ \frac{d(t) (\max \{ \frac{a^{u}}{b^{l}},m^{u} \}-m(t) )}{a_{1}(t)-m(t)} \biggr]. \end{aligned} $$
(4.11)

Integrating Eq. (4.11) from \(t_{1}\) to t, it follows that

$$ \begin{aligned} y(t)&\leq y(t_{1})e^{\int _{t_{1}}^{t} [-r_{1}(s)+ \frac{d(s) (\max \{ \frac{a^{u}}{b^{l}},m^{u} \}-m(s) )}{a_{1}(s)-m(s)} ] \,\mathrm{d}s} \\ &\leq y(t_{1})e^{\int _{t_{1}}^{t} [-r_{1}^{l}+ \frac{d^{u} (\max \{ \frac{a^{u}}{b^{l}},m^{u} \}-m^{l} )}{a_{1}^{l}-m^{u}} ] \,\mathrm{d}s}. \end{aligned} $$

Let \(t\rightarrow \infty \), then

$$ \lim_{t\rightarrow \infty }y(t)=0. $$

This completes the proof of Theorem 4.1. □

5 Numeric simulations

Now let us consider the following three examples.

Example 5.1

$$ \begin{gathered} \dot{x}(t)=x(t)\bigl[11+\cos t-3x(t)\bigr]- \frac{5(x(t)-0.5)y(t)}{x(t)+2}, \\ \dot{y}(t)=y(t) \biggl[-0.2-4y(t)+ \frac{(4+0.5\sin t)(x(t)-0.5)}{x(t)+2} \biggr]. \end{gathered} $$
(5.1)

In system (5.1), corresponding to system (1.4), we assume that \(a(t)=11+\cos t\), \(b(t)=3\), \(c(t)=5\), \(m(t)=0.5\), \(a_{1}(t)=2.5\), \(r_{1}(t)=0.2\), \(k(t)=4\) and \(d(t)=4+0.5\sin t\), then

$$ \begin{gathered} M_{1}= \frac{a^{u}}{b^{l}}=4,\qquad M_{2}= \frac{d^{u}M_{1}}{k^{l}(a_{1}^{l}-m^{u})}=2.25,\\ m_{1}= \frac{a^{l}(a_{1}^{l}-m^{u})-c^{u}M_{2}}{b^{u}(a_{1}^{l}-m^{u})} \approx 1.46, \end{gathered} $$

and

$$\begin{aligned}& a_{1}^{l}-m^{u}=2>0,\qquad \frac{c^{u}M_{2}}{a_{1}^{l}-m^{u}} \approx 5.625< a^{l}=10, \\& \frac{d^{l}(m_{1}-m^{u})}{a_{1}^{u}+M_{1}}\approx 0.21>r_{1}^{u}=0.2, \qquad m_{1}=1.46>m^{u}=0.5, \end{aligned}$$

then condition (2.1) and (2.2) are satisfied. According to Lemma 2.2, the system (1.4) is permanent. Numerical simulation (see Fig. 1 and Fig. 2) also supports this conclusion.

Figure 1
figure 1

Dynamic behaviors of the prey population of the system (5.1), with the initial condition condition \((x(0),y(0)) =(6,2),(2,0.5),(0.5,1)\) and \((4,3)\), respectively

Figure 2
figure 2

Dynamic behaviors of the predator population of the system (5.1), with the initial condition condition \((x(0),y(0)) =(6,2),(2,0.5),(0.5,1)\) and \((4,3)\), respectively

Example 5.2

$$ \begin{gathered} \dot{x}(t)=x(t)\bigl[5+\cos t-3x(t)\bigr]- \frac{5(x(t)-2.3)y(t)}{x(t)+0.2}, \\ \dot{y}(t)=y(t) \biggl[-0.2-4y(t)+ \frac{(4+0.5\sin t)(x(t)-2.3)}{x(t)+0.2} \biggr]. \end{gathered} $$
(5.2)

In system (5.2), corresponding to system (1.4), we assume that \(a(t)=5+\cos t\), \(b(t)=3\), \(c(t)=5\), \(m(t)=2.3\), \(a_{1}(t)=2.5\), \(r_{1}(t)=0.2\), \(k(t)=4\) and \(d(t)=4+0.5\sin t\), then

$$ M_{1}= \frac{a^{u}}{b^{l}}=2 $$

and

$$ a_{1}^{l}-m^{u}=0.2>0,\qquad -r_{1}^{l}+ \frac{b^{u} (\max \{ \frac{a^{u}}{b^{l}},m^{u} \}-m^{l} )}{a_{1}^{l}-m^{u}}=-0.2< 0 $$

then condition (2.1) and (4.3) are satisfied. According to Theorem 4.1, the predator population of system (1.4) will go extinct. Here, \(m^{u}=2.3>M_{1}=2\). It follows that, when the maximum capacity of the refuge is larger than the maximum of the prey population, the predator population will go extinct. Numerical simulation (see Fig. 3) also supports this conclusion.

Figure 3
figure 3

The extinction of the predator population of the system (5.2), with the initial condition condition \((x(0),y(0)) =(6,2),(2,0.5),(0.5,0.1)\) and \((4,0.3)\), respectively

Example 5.3

$$ \begin{aligned} &\dot{x}(t)=x(t)\bigl[5+\cos t-3x(t)\bigr]- \frac{5(x(t)-1.5)y(t)}{x(t)+1}, \\ &\dot{y}(t)=y(t) \biggl[-1.7-4y(t)+ \frac{(4+0.5\sin t)(x(t)-1.5)}{x(t)+1} \biggr]. \end{aligned} $$
(5.3)

In system (5.3), corresponding to system (1.4), we assume that \(a(t)=5+\cos t\), \(b(t)=3\), \(c(t)=5\), \(m(t)=1.5\), \(a_{1}(t)=2.5\), \(r_{1}(t)=1.7\), \(k(t)=4\) and \(d(t)=4+0.5\sin t\), then

$$ M_{1}= \frac{a^{u}}{b^{l}}=2 $$

and

$$ a_{1}^{l}-m^{u}=1>0,\qquad -r_{1}^{l}+ \frac{b^{u} (\max \{ \frac{a^{u}}{b^{l}},m^{u} \}-m^{l} )}{a_{1}^{l}-m^{u}}=-0.2< 0, $$

then condition (2.1) and (4.3) are satisfied. According to Theorem 4.1, the predator population of system (1.4) will go extinct. Here, \(m^{u}=1.5< M_{1}=2\). It follows that, when the maximum capacity of refuge is lower than the maximum of the prey population, the predator population will go extinct if the per capita death rate of predator is high enough. Numerical simulation (see Fig. 4) also supports this conclusion.

Figure 4
figure 4

The extinction of the predator population of the system (5.3), with the initial condition condition \((x(0),y(0)) =(6,1),(4,0.6),(0.5,0.2)\) and \((1,0.05)\) respectively

6 Conclusion

In this paper, we consider a nonautonomous predator–prey model with Holling type II schemes and a prey refuge. Firstly, by applying the comparison theorem of differential equations, a set of conditions that ensure the permanence of the system is obtained. Secondly, by constructing a suitable Lyapunov function, sufficient conditions that guarantee the permanence and global stability of the system are investigated. Lastly, by applying the oscillation theory and the comparison theorem of differential equations, a set of sufficient conditions that guarantee the extinction of the predator of the system is obtained.

Condition (4.3) implies two situations:

(1) When the maximum capacity of refuge is larger than the maximum of the prey population, the predator population will go extinct.

(2) When the maximum capacity of refuge is lower than the maximum of the prey population, as long as per capita death rate of predator is large enough, then the predator population will go extinct.