ABSTRACT This paper deals with the twoâplayer zeroâsum game problem, which is a bounded âgain robust control problem. Finding an analytical solution to the complex HamiltonâJacobiâIssacs (HJI) equation is a challenging task. Hence, a novel Qâlearning algorithm for unknown continuousâtime (CT) affineâinâinputs nonlinear systems is proposed for generating an approximate solution to the HJI equation, which is valid in a local domain due to the use of a local approximator, that is, a Neural Network (NN) structure. The approach is modelâfree and does not require the knowledge of system drift dynamics, and input and disturbance gains. The algorithm learns online from measurements of state variables in real time. To generate the local approximate solution of the HJI equation for the twoâplayer zeroâsum game problem for nonlinear systems, the proposed nonâiterative algorithm requires only a single critic NN instead of the commonly used triple NN approximator structure. A persistence of excitation condition is required to guarantee Uniformly Ultimately Boundedness (UUB) and convergence to the optimal solution. The effectiveness of the proposed Qâlearning approach for the twoâplayer zeroâsum game problem is demonstrated via simulations of a linear Fâ16 aircraft plant and a highly complex nonlinear system. Proof of closedâloop system stability is provided using Lyapunov Analysis, and convergence of the approximate solution to the true saddleâpoint solution is guaranteed in a UUBâsense.
In this paper, we formulate inverse reinforcement learning (IRL) as an expert-learner interaction whereby the optimal performance intent of an expert or target agent is unknown to a learner agent. The learner observes the states and controls of the expert and hence seeks to reconstruct the expert's cost function intent and thus mimics the expert's optimal response. Next, we add non-cooperative disturbances that seek to disrupt the learning and stability of the learner agent. This leads to the formulation of a new interaction we call zero-sum game IRL. We develop a framework to solve the zero-sum game IRL problem that is a modified extension of RL policy iteration (PI) to allow unknown expert performance intentions to be computed and non-cooperative disturbances to be rejected. The framework has two parts: a value function and control action update based on an extension of PI, and a cost function update based on standard inverse optimal control. Then, we eventually develop an off-policy IRL algorithm that does not require knowledge of the expert and learner agent dynamics and performs single-loop learning. Rigorous proofs and analyses are given. Finally, simulation experiments are presented to show the effectiveness of the new approach.
Sayan Basu Roy, Shubhendu Bhasin, Indra Narayan Kar
Convergence of controller parameters in standard model reference adaptive control (MRAC) requires the system states to be persistently exciting (PE), a restrictive condition to be verified online. A recent data-driven approach, concurrent learning, uses information-rich past data concurrently with the standard parameter update laws to guarantee parameter convergence without the need of the PE condition. This method guarantees exponential convergence of both the tracking and the controller parameter estimation errors to zero, whereas, the classical MRAC merely ensures asymptotic convergence of tracking error to zero. However, the method requires knowledge of the state derivative, at least at the time instances when the state values are stored in memory. The method further assumes knowledge of the control allocation matrix. This paper addresses these limitations by using a memory-based finite-time system identifier in conjunction with a data-driven approach, leading to convergence of both the tracking and the controller parameter estimation errors without the PE condition and knowledge of the system matrices and the state derivative. A Lyapunov based stability proof is included to justify the validity of the proposed data-driven approach. Simulation results demonstrate the efficacy of the suggested method.