1 Introduction

It is well known that the instability of the solutions is a very important problem in the theory and applications of differential equations. For example, if the solutions of a differential equation describing a dynamical system or of any differential equation under consideration are known in closed form, then one can determine the instability properties of the system or the solutions of the differential equation taken under consideration, appealing directly the definition of the instability, which will be introduced hereinafter. In addition, ideally, one would like to compute clearly all solutions of every differential equation or system of differential equations. However, as we know, there are actually very few equations (beyond linear equations with constant coefficients – and even there are difficulties if the order of the equation or system is high) for which we can do this. That is, in general, it is not possible to find the solution of all linear and nonlinear differential equations, except numerically. Moreover, finding solutions becomes more difficult for delay differential equations rather than differential equations without delay. Therefore, it is very important to get information about the qualitative behavior of solutions of delay differential equations when there is no analytical expression for the solutions. In the literature, specific methods have been developed to obtain information on the qualitative behavior of solutions of differential equations when there is no analytical expression for the solutions. One of them is known as Lyapunov's second (or direct) method. More than 100 years ago, the world-famous mathematician Lyapunov established this method to study stability problems. Today, this method is widely recognized as an excellent tool not only in the study of differential equations but also in the theory of control systems, dynamical systems, system with time lag, power system analysis, and so on. It is worth mentioning that the expressions of Lyapunov functionals are very complicated and hard to construct. The method is an interesting and fruitful technique to determine the instability and the stability behaviors of solutions of linear and non-linear differential equations. This technique has gained increasing significance and has given impetus for modern development of instability and stability theory of differential equations. An apparent advantage of this method is that the instability and the stability in the large can be obtained without any prior knowledge of solutions. That is, the method yields instability and stability information directly, without solving the differential equation. The chief characteristic of the method requires the construction of the scalar function or functional for the equation under study. Unfortunately, it is sometimes very difficult to find a proper Lyapunov function or functional for a given differential equation.

However, from then on, the Lyapunov's direct method was also widely used and is still being employed to study the instability of solutions of ordinary differential equations and functional differential equations of fifth order, see e.g. Tunç [1-10], Li and Duan [11], Ezeilo [12-14], Li and Yu [15], Sadek [16], Sun and Hou [17], Tiryaki [18], Tunç and Erdogan [19], Tunç and Karta [20], and Tunç and Şevli [21] and the references therein. Besides, it is worth mentioning that to the best of our knowledge, the instability of the solutions of certain functional differential equations of the fifth order has been discussed in the literature, recently (see [1-7]).

In this case, it is worthwhile to continue the investigation of the instability of the solutions of functional differential equations of fifth order.

It should be noted that the author in ([1,2,5,6]) considered the functional differential equations of the fifth order

\[x^{(5)}(t) + \psi_1(x''(t))x'''(t) + \phi(x(t), x(t-r), \dots, x^{(4)}(t), x^{(4)}(t-r))x''(t)\] \[+ \theta_1(x'(t)) + f_1(x(t-r)) = 0,\] \[x^{(5)}(t) + a_1x^{(4)}(t) + k(x(t), x'(t), x''(t), x'''(t), x^{(4)}(t))x'''(t) + g(x'(t))x''(t)\] \[+ h(x(t), x'(t), x''(t), x''(t), x^{(4)}(t)) + f(x(t-r)) = 0,\] \[x^{(5)}(t) + a_1x^{(4)}(t) + a_2x'''(t) + a_3x''(t) + a_4x'(t) + f(x(t-r)) = 0\] and

\[x^{(5)}(t) + a_1 x^{(4)}(t) + a_2 x'''(t) + g(x'(t))x''(t)\]
+ \(h(x(t - \tau(t)), x'(t - \tau(t)), \dots, x^{(4)}(t - \tau(t))) + f(x(t - \tau(t))) = 0,\)

respectively. Then, the author obtained some sufficient conditions ensuring that the zero solution of these equations is unstable by defining some appropriate Lyapunov-Krasovskii functionals.

Besides, Li and Duan [11] considered the equation

\[x^{(5)}(t) + a_5 x^{(4)}(t) + a_4 x'''(t) + f_3(x(t), x'(t), x''(t), x'''(t), x'''(t)) x''(t) + f_2(x(t), x'(t), x''(t), x'''(t), x'''(t), x'''(t) + f_1(x(t)) = 0.\] (1)

By using the Četaev's instability theorem, they obtained sufficient conditions for the zero solution of Eq. (1) to be unstable, see LaSalle and Lefschetz [22].

In this paper, instead of Eq. (1), we consider fifth order nonlinear delay differential equation

\[\begin{split} x^{(5)}(t) + a_5 x^{(4)}(t) + a_4 x'''(t) \\ + f_3(x(t-r), x'(t-r), x''(t-r), x'''(t-r), x'''(t-r), x'''(t-r)) x''(t) \\ + f_2(x(t-r), x'(t-r), x''(t-r), x'''(t-r), x'''(t-r), x'''(t-r)) x'(t) \\ + f_1(x(t-r)) &= 0. \end{split}\]

We write Eq. (2) as the system

\[x'_{1}(t) = x_{2}(t), \quad x'_{2}(t) = x_{3}(t), \quad x'_{3}(t) = x_{4}(t), \quad x'_{4}(t) = x_{5}(t),\] \[x'_{5}(t) = -a_{5}x_{5}(t) - a_{4}x_{4}(t) - f_{3}(x_{1}(t-r), \dots, x_{5}(t-r))x_{3}(t)\] \[-f_{2}(x_{1}(t-r), \dots, x_{5}(t-r))x_{2}(t)\] \[-f_{1}(x_{1}(t)) + \int_{t-r}^{t} f'_{1}(x_{1}(s))x_{2}(s)ds,\] (3)

where \(a_5\), \(a_4\) and r(>0) are constants, r is fixed delay, the primes in Eq. (2) denote differentiation with respect to t, \(t \in \Re^+ = [0, \infty)\); \(f_1\), \(f_2\) and \(f_3\) are continuous functions on \(\Re\), \(\Re^5\) and \(\Re^5\), respectively, and with \(f_1(0) = 0\). The continuity of the functions \(f_1\), \(f_2\) and \(f_3\) is a sufficient condition for the existence of the solution of Eq. (2) (see [23, pp.14]). It is also assumed as basic that the functions \(f_1\), \(f_2\) and \(f_3\) satisfy a Lipschitz condition in their respective arguments. Hence, the uniqueness of the solutions of Eq. (2) is guaranteed (see [23, pp.15]). We also assume throughout what follows that \(f_1\) is differentiable, and \(x_1(t),...,x_5(t)\) are abbreviated as \(x_1,...,x_5\), respectively.

It should also be noted that, in reality, many systems have the property of aftereffect, i.e. the future states depend not only on the present, but also on the past. Therefore, the investigation of the instability of the solutions of delay differential equations of higher order is very considerable in the literature. The purpose of this paper is to present a new result on the instability of the zero solution of Eq. (2). Our method relies on the Lyapunov-Krasovskii functional approach (see [24]). This method permits us to obtain a new result on Eq. (2) under quite general assumptions on the nonlinearities. The obtained result improves and enhances the result in Li and Duan [11, Theorem 5] for the case without delay to the case with delay. Here, by defining an appropriate Lyapunov-Krasovskii functional, we carry out our purpose. It should be noted that the result to be established here is different from that in Tunç ([1], [2], [5], [6]) and the literature.

It is worth mentioning that for some recent works on the qualitative behaviors of solutions, we can also refer to Zhu and Shong [25] and Zhu, et al. [26].

In the following theorems, we give a basic idea of the method about the instability of solutions of ordinary and delay differential equations. The following theorem is due to the Russian mathematician N.G. Četaev (see LaSalle and Lefschetz [22]).

Theorem A (Instability Theorem of Četaev).

Let \(\Omega\) be a neighborhood of the origin. Let there be given a function V(x) and region \(\Omega_1\) in \(\Omega\) with the following properties:

• 1. V(x) has continuous first partial derivatives in \(\Omega_1\).
• 2. V(x) and \(\dot{V}(x)\) are positive in \(\Omega_1\).

• 3. At the boundary points of \(\Omega_1\) inside \(\Omega\), V(x) = 0.
• 4. The origin is a boundary point of \(\Omega_1\).

Under these conditions the origin is unstable.

Let \(r \ge 0\) be given, and let \(C = C([-r,0], \Re^n)\) with

\[\|\varphi\| = \max_{s \in S \setminus S} |\varphi(s)|, \ \varphi \in C.\]

For H > 0 define \(C_H \subset C\) by

\[C_H = \{ \varphi \in C : \| \varphi \| < H \}.\]

If \(x:[-r,A) \to \Re^n\) is continuous, \(0 < A \le \infty\), then, for each t in [0,A), \(x_t\) in C is defined by

\[x_t(s) = x(t+s), -r \le s \le 0, \ t \ge 0.\]

Let G be an open subset of C and consider the general autonomous delay differential system with finite delay

\[\dot{x} = F(x_t), \quad x_t = x(t+\theta), \quad -r \le \theta \le 0, \quad t \ge 0,\] where \(F: G \to \Re^n\) is continuous and maps closed and bounded sets into bounded sets and F(0) = 0. It follows from the conditions on F that each initial value problem

\[\dot{x} = F(x_t), \ x_0 = \phi \in G\]

has a unique solution defined on some interval [0, A), \(0 < A \le \infty\). This solution will be denoted by \(x(\phi)(.)\) so that \(x_0(\phi) = \phi\).

Definition. The zero solution of \(\dot{x} = F(x_t)\) is stable if for each \(\varepsilon > 0\) there exists \(\delta = \delta(\varepsilon) > 0\) such that \(\|\phi\| < \delta\) implies that \(|x(\phi)(t)| < \varepsilon\) for all \(t \ge 0\). The zero solution is said to be unstable if it is not stable.

2 Statement of Main Result

In this section, we will state our main result. The main result is the following theorem.

Theorem. Assume that there exist constants \(a_1\) (>0), \(a_4\) and \(\delta\) such that the following conditions hold:

\[f_1(0) = 0\], \(f_1(x_1) \neq 0\), \((x_1 \neq 0)\), \(a_4 \leq -1\), \(|f_1'(x_1)| \leq a_1\) for all \(x_1\)

and

\[f_2(x_1(t-r),...,x_s(t-r)) - f_3^2(x_1(t-r),...,x_s(t-r)) > \delta > 0\] for all \(x_1(t-r),...,x_5(t-r)\).

If

\[r < \frac{\delta}{a_1}\], then the zero solution of Eq. (2) is unstable.

Remark. It is clear that Eq. (2) has the zero solution since \(f_1(0) = 0\).

It should be noted that the proof of the main result is based on the instability criteria of Krasovskii [24]. Because of these criteria, it is necessary to show here that there exists a Lyapunov-Krasovskii functional \(V = V(x_{1t},...,x_{5t})\) that has Krasovskii properties, say \((P_1)\), \((P_2)\) and \((P_3)\):

• \((P_1)\) In every neighborhood of (0,0,0,0,0), there exists a point \((\xi_1,...,\xi_5)\) such that \(V(\xi_1,...,\xi_5)>0\),
• \((P_2)\) the time derivative \(\frac{d}{dt}V(x_{1t},...,x_{5t})\) along solution paths of (3) is positive semi-definite,
• \((P_3)\) the only solution \((x_1,...,x_5) = (x_1(t),...,x_5(t))\) of (3) which satisfies \(\frac{d}{dt}V(x_{1t},...,x_{5t}) = 0, (t > 0)\), is the trivial solution (0,0,0,0,0).

Proof. We define a Lyapunov-Krasovskii functional \(V = V(x_{1t},...,x_{5t})\):

\[V = -x_2x_5 + x_3x_4 - a_5x_2x_4 + \frac{1}{2}a_5x_3^2 - a_4x_2x_3 - \int_0^{x_1} f_1(s)ds\]

\[-\mu \int_{-r}^{0} \int_{t+s}^{t} x_2^2(\theta) d\theta ds, \tag{4}\] where \(a_5(>0)\), \(a_4\) are constants, s is a real variable such that the integral \(\int_{-r}^{0} \int_{t+s}^{t} x_2^2(\theta) d\theta ds\) is non-negative, and \(\mu\) is positive constant, which will be determined later in the proof.

Hence, it is clear from the definition of V that

\[V(0,0,0,0,0) = 0\] and

\[V(0,0,\varepsilon,\varepsilon,0) = \varepsilon^2 + \frac{1}{2}a_5\varepsilon^2 > 0\] for each arbitrary sufficiently small \(\varepsilon\) so that every neighborhood of the origin in the \((x_1,...,x_5)-\) space contains the points \((\xi_1,...,\xi_5)\) such that \(V(\xi_1,...,\xi_5)>0\).

Let

\[(x_1,...,x_5) = (x_1(t),...,x_5(t))\] be an arbitrary solution of (3). By elementary differentiation, the time derivative of the Lyapunov functional V in (4) along the solutions of (3) yields

\[\frac{d}{dt}V(x_{1t},...,x_{5t}) = x_4^2 - a_4 x_3^2 + f_3(x_1(t-r),...,x_5(t-r))x_2 x_3\] \[+ f_2(x_1(t-r),...,x_5(t-r))x_2^2\] \[-x_2 \int_0^t f_1'(x_1(s))x_2(s)ds - \mu r x_2^2 + \mu \int_0^t x_2^2(s)ds.\]

Using the assumption \(|f_1'(x_1)| \le a_1\) of the theorem and the estimate \(|mn| \le m^2 + n^2\), we get

\[-x_2(t)\int_{t-r}^t f_1'(x_1(s))x_2(s)ds \ge -|x_2(t)|\int_{t-r}^t |f_1'(x_1(s))| |x_2(s)|ds\]

\[\geq -\frac{1}{2} \int_{t-r}^{t} |f_1'(x_1(s))| (x_2^2(t) + x_2^2(s)) ds\] \[\geq -\frac{a_1}{2} \int_{t-r}^{t} (x_2^2(t) + x_2^2(s)) ds\] \[= -\frac{1}{2} a_1 x_2^2(t) r - \frac{1}{2} a_1 \int_{t-r}^{t} x_2^2(s) ds.\]

Hence, it follows that

\[\frac{d}{dt}V(x_{1t},...,x_{5t}) \ge x_4^2 + \frac{3}{4}x_3^2 + \left(\frac{x_3}{2} + f_3(x_1(t-r),...,x_5(t-r))x_2\right)^2 \\ + \left\{f_2(x_1(t-r),...,x_5(t-r) - (\mu + \frac{1}{2}a_1)r\right\}x_2^2 \\ - \left\{f_3^2(x_1(t-r),...,x_5(t-r))\right\}x_2^2 \\ + \left(\mu - \frac{1}{2}a_1\right)\int_{t-r}^{t} x_2^2(s)ds.\]

Let \(\mu = \frac{1}{2}a_1\). Using the assumptions of the theorem, we get

\[\frac{d}{dt}V(x_{1t},...,x_{5t}) \ge x_4^2 + \frac{3}{4}x_3^2 + f_2(x_1(t-r),...,x_5(t-r))x_2^2 - \{f_3^2(x_1(t-r),...,x_5(t-r)) - a_1r\}x_2^2 \ge x_4^2 + \frac{3}{4}x_3^2 + (\delta - a_1r)x_2^2.\]

If \(r < \frac{\delta}{a_1}\), then we have for a positive constant k that

\[\frac{d}{dt}V(x_{1t},...,x_{5t}) \ge x_4^2 + \frac{3}{4}x_3^2 + kx_2^2 > 0.\]

Then, the Lyapunov-Krasovskii functional V satisfies the property \((P_2)\).

Besides, \[\frac{d}{dt}V(x_{1t},...,x_{5t}) = 0\] if and only if

\[x_2 = x_3 = x_4 = x_5 = 0.\]

The substitution of the above estimate in (3) gives \(f_1(\xi_1) = 0\). This result implies that \(\xi_1 = 0\) by the assumption \(f_1(x_1) \neq 0\), \((x_1 \neq 0)\). Hence, \(\dot{V} = 0\) \((t \geq 0)\) so that

\[x_1 = x_2 = x_3 = x_4 = x_5 = 0, (t \ge 0).\]

Thus, the functional V has all the requisite Krasovskii [24] properties subject to the conditions of the theorem, which now follows. By the above discussion, we can conclude that the zero solution of Eq. (2) is unstable. The proof for the theorem is complete.

3 Conclusion

A non-linear functional differential equation of fifth order with constant delay is considered. Based on the Krasovskii properties, the instability of the zero solution of this equation is discussed. In proving our result, we employ the Lyapunov-Krasovskii functional approach by defining a new Lyapunov-Krasovskii functional. Our result improves some known results from the scalar case to the vectorial case.

Acknowledgements

The author of this paper would like to expresses his sincere thanks to the anonymous referees for their many helpful comments, corrections and suggestions on the paper.