The control of chaotic systems has been the focus of control research in the past decade.^{[1– 7]} For most control methods of chaotic systems, such as impulsive control methods^{[2– 4, 6]} and adaptive synchronization control methods, ^{[8– 10]} only the stability properties of the chaotic systems are considered. The optimality property is an important performance of chaotic control systems. As is well known, dynamic programming is a very useful tool in solving optimal control problems.^[11] However, due to the difficulties related to solving the time-varying Hamilton– Jacobi– Bellman (HJB) equations, the closed-loop optimal feedback control can hardly be obtained. Approximate optimal solutions of the optimal problem have consequently attracted much attention.^{[12– 16]} Among those approximate approaches, the adaptive dynamic programming (ADP) algorithm, proposed by Werbos, ^{[17, 18]} has played an important role in seeking the approximate solutions of dynamic programming problems.^{[19– 21]} There are several synonyms used for ADP, including adaptive critic designs, ^[22] adaptive dynamic programming, ^{[23– 27]} approximate dynamic programming, ^[28] neural dynamic programming, ^[29] and reinforcement learning.^[30] In Refs. [22] and [28], ADP approaches were classified into several main schemes, which are heuristic dynamic programming (HDP), action-dependent HDP (ADHDP), dual heuristic dynamic programming (DHP), action-dependent DHP (ADDHP), which is also called Q-learning, ^[31] globalized DHP (GDHP), and action-dependent GDHP (ADGDHP). Iterative methods are also used in ADP to obtain the solution of the HJB equation indirectly and they have received increasing attention.^{[32– 40]} There are two main iterative ADP algorithms, ^[41] which are based on value and policy iterations, respectively.

Value iteration algorithms for the optimal control of discrete-time nonlinear systems were given in Ref. [42]. Al-Tamimi and Lewis^[43] studied the deterministic discrete-time affine nonlinear systems and a value iteration algorithm, which is referred to as HDP and proposed for finding the optimal control law. Starting from J^[0](x_k) ≡ 0, the value iteration algorithm is updated by

for i = 0, 1, 2, … c, where x_{k+ 1} = f(x_k) + g(x_k)u^[i](x_k). In Ref. [43], it was proved that J^[i](x_k) is a nondecreasing sequence and bounded, and hence converges to J* (x_k). Zhang et al. applied a value iteration algorithm for optimal tracking problems.^[36] Song et al. successfully implemented value iteration algorithms to find the optimal tracking control law for Hé non chaotic systems.^[44] For most previous value iteration algorithms of ADP, the initial performance index function is chosen as zero.^{[36, 43, 44]} For each of the iterative controls u^[i](x_k), i = 0, 1, … , the stability of the system cannot be guaranteed. This means that only the converged u* (x_k) can be used to control the nonlinear system, and all the iterative controls u^[i](x_k), i = 0, 1, … may be invalid. So the computation efficiency of the value iteration ADP method is very low.

Policy iteration algorithms for the optimal control of continuous-time systems with continuous state and action spaces were given in Refs. [12] and [32]. In Ref. [12], Murray et al. proved that for continuous-time affine nonlinear systems, each of the iterative controls stabilizes the nonlinear systems by using the policy iteration algorithms. This is the great merit of the policy iteration algorithms. However, almost all of the discussions on the policy iteration algorithms are for continuous-time control systems.^[35] In Ref. [45], the policy iteration algorithm for discrete-time systems was displayed and the stability and convergence properties were discussed. To the best of our knowledge, there is still no discussion focused on the policy iteration algorithms for the optimal tracking controls of discrete-time chaotic systems. This motives our research.

In this paper, inspired by Ref. [45], we develop a policy iteration ADP algorithm for the first time to solve the optimal tracking controls of discrete-time chaotic systems. First, by system transformations, the optimal tracking problem is transformed into an optimal regulation problem. Next, a policy iteration algorithm for discrete-time chaotic systems is described. Then, the convergence and admissibility properties of the developed policy iteration algorithm are presented, which show that the transformed chaotic system can be stabilized under an arbitrary iterative control law and simultaneously guarantee the iterative performance index function to converge to the optimum. By employing neural networks, the developed policy iteration algorithm is implemented. To justify the effectiveness of the developed algorithm, the simulation results using the policy iteration algorithm are compared with the results obtained by the traditional value iteration algorithm.

This paper is organized as follows. In Section 2, we present the problem formulation and preliminaries. In Section 3, the optimal chaotic tracking control scheme based on the policy iteration ADP algorithm is given, where the corresponding convergence and admissibility properties will also be presented. In Section 4, a simulation example is given to demonstrate the effectiveness of the developed optimal tracking control scheme. In Section 5, the conclusion is given.

Consider the following MIMO chaotic dynamic system:

where x_k = [x_{1, k}, … , x_{n, k}]^T ∈ ℝ ⁿ is the system state vector which is assumed to be available from measurement. Let u_k = [u_{1, k}, … , u_{m, k}]^T∈ ℝ ^m be the control input and g_ij (i = 1, … , n, j = 1, … , m) be the constant control gain. If we denote f(x_k) = [f₁(x_k), … , f_n(x_k)]^T and , then the chaotic dynamic system (2) can be rewritten as

It should be pointed out that system (3) denotes a large class of chaotic systems, such as the Hé non systems^[46] and the new discrete chaotic system proposed in Ref. [47]. In an optimal tracking problem, the control objective is to design optimal control u(x_k) for system (2) such that the system state x_k tracks the specified desired trajectory η _k ∈ ℝ ⁿ, k = 0, 1, … , where η _k satisfies

and σ is a given function. We define the tracking error as

and define the following quadratic performance index:

where Q ∈ ℝ ^{n× n} and R ∈ ℝ ^{m× m} are positive definite matrices and u₀ = (u₀, u₁, … ). Let

be the utility function, where w_k = u_k − u_{e, k}. Let u_{e, k} denote the expected control introduced for analytical purpose, which can be given as

where g^{− 1}(η _k)g(η _k) = I and I ∈ ℝ ^{m× m} is the identity matrix. Combining Eqs. (2) and (3), we obtain

We will study optimal tracking control problems for system (2). The goal is to find an optimal tracking control scheme which tracks the desired trajectory η _k and simultaneously minimizes the performance index function (6). The optimal performance index function is defined as

where w_k = (w_k, w_{k+ 1}, … ). According to Bellman’ s principle of optimality, J* (z_k) satisfies the discrete-time HJB equation

Then, the law of optimal single control vector can be expressed as

Hence, the HJB equation (11) can be written as

From Eqs. (4)– (13), we can see that the optimal tracking problem for chaotic system (3) is transformed into an optimal regulation problem for chaotic system (9). The objective of this paper is to construct an optimal tracking controller such that the chaotic system state x_k tracks the reference signal η _k, i.e., z_k→ 0, as k→ 0. For convenience of analysis, the results of this paper are based on the following assumptions.

Assumption 1 The system (9) is controllable and the function 𝓕 (z_k, w_k) is Lipschitz continuous for ∀ z_k, w_k.

Assumption 2 The system state z_k = 0 is an equilibrium state of system (9) under the control v = 0, i.e., F(0, 0) = 0.

Assumption 3The feed-back control w_k = w(z_k) satisfies w(z_k) = 0 for z_k = 0.

Assumption 4 The utility function U(z_k, w_k) is a continuous positive definite function of z_k and w_k.

Generally, J* (z_k) is unknown before the control error w_k ∈ ℝ ^m is considered. If we adopt the traditional dynamic programming method to obtain the optimal performance index function, then we have to face “ the curse of dimensionality” . This makes the optimal control nearly impossible to obtain by the HJB equation. So, in the next section, an effective policy iteration ADP algorithm will be developed to obtain the optimal chaotic tracking control iteratively.

In this section, a new discrete-time policy iteration algorithm is developed to obtain the optimal tracking control law for chaotic systems (2). The goal of the present iterative ADP algorithm is to construct an optimal control law w* (z_k), k = 0, 1, … , which moves an arbitrary initial state x₀ to the desired trajectory η _k, and simultaneously minimizes the performance index function. Convergence and admissibility properties will be analyzed.

To evaluate the performance of our policy iteration algorithm, we choose an example with quadratic utility functions for numerical experiments. Consider the following chaotic system:^[4]

where h(x₁) = m₁x₁ + (m₀ − m₁)(| x₁ + θ ₃| − | x₁ − θ ₃| )/2, θ ₁ = 9, θ ₂ = 14.28, θ ₃ = 1, m₀ = − 1/7, and m₁ = 2/7. The state trajectory of the chaotic system is given in Fig. 1. According to Euler’ s discretization method, the continuous time chaotic system can be represented as follows:

where Δ T = 0.1. Let the desired orbit be η _k = [sin(k), 0.5cos(k), 0.6sin(k)]^T and the initial state be selected as [1, − 1, 1.5]^T. The utility function is defined as , where Q = R = I₃. Neural networks are used to implement the developed policy iteration algorithm.

Fig. 1.

	Figure Option ViewDownloadNew Window
	Fig. 1. The chaotic attractor of system (54).

The critic network and the action network are chosen as three-layer BP neural networks with the structures of 3– 10– 1 and 3– 10– 3, respectively. For each iteration step, the critic network and the action network are trained for 1000 steps using a learning rate of α = 0.02 so that the neural network training error becomes less than 10^{− 5}. The training methods of the neural networks have been described in Refs. [30] and [45], which are omitted here. The performance index function is shown in Fig. 2. The system states are shown in Fig. 3. The state errors are given in Fig. 4. The iterative controls are shown in Fig. 5. In the simulation results, we let “ initial” and “ limiting” denote the initial iteration and the limiting iteration, respectively.

Fig. 2.

	Figure Option ViewDownloadNew Window
	Fig. 2. The performance index function trajectory in the policy iteration algorithm.

Fig. 3.

	Figure Option ViewDownloadNew Window
	Fig. 3. The trajectories of system states x1, x2, and x3 in the policy iteration algorithm.

Fig. 4.

	Figure Option ViewDownloadNew Window
	Fig. 4. The trajectories of tracking errors z1, z2, and z3 in the policy iteration algorithm.

Fig. 5.

	Figure Option ViewDownloadNew Window
	Fig. 5. The trajectories of controls u1, u2, and u3 in the policy iteration algorithm.

From the simulation results, we can see that the iterative performance index function J^[i](z_k) is monotonically non-increasing and converges to the optimal performance index function. For ∀ i = 0, 1, … , the regulation system (9) is stable under the iterative control law w^[i](z_k). To show the effectiveness of the developed algorithm, the results of our developed policy iteration algorithm are compared with the results using a value iteration algorithm.^[36] Let the value iteration algorithm be implemented by Eq. (1), where the initial performance index function J^[0](z_k) ≡ 0. All of the parameters are set the same as those for implementing the polity iteration algorithm. The performance index function is shown in Fig. 6. The system states are shown in Fig. 7. The iterative controls are shown in Fig. 8.

Fig. 6.

	Figure Option ViewDownloadNew Window
	Fig. 6. The performance index function trajectory in the value iteration algorithm.

Fig. 7.

	Figure Option ViewDownloadNew Window
	Fig. 7. The trajectories of system states x1, x2, and x3 in the value iteration algorithm.

Fig. 8.

	Figure Option ViewDownloadNew Window
	Fig. 8. The trajectories of controls u1, u2, and u3 in the value iteration algorithm.

From Fig. 6, we can see that the iterative performance index function is monotonically non-decreasing and converges to the optimum. The convergence properties of value and policy iteration algorithms are inherently different. On the other hand, from Figs. 7 and 8, we can see that there exit unstable system states and control laws in the value iteration algorithm. From Figs. 3 and 5, we can see that the chaotic system (54) can track the desired trajectories with the policy iteration algorithm, which illustrates the effectiveness of the developed algorithm.

We have developed an optimal tracking control method for chaotic systems by policy iteration. By system transformations, the optimal tracking problem is transformed into an optimal regulation one. Next, the policy iteration algorithm for the transformed chaotic systems are presented. The convergence and admissibility properties of the developed policy iteration algorithm are presented, which show that the transformed chaotic system can be stabilized under the present iterative control law and the iterative performance index function simultaneously converges to the optimum. Finally, the effectiveness of the developed optimal tracking control scheme for chaotic systems is verified by a simulation.