This article addresses the challenging problem of constrained optimal control for humanâmachine systems subject to external disturbances and the bounded rationality of the human operator. To this end, a novel game-theoretic framework is proposed. Unlike monolithic game formulations, the framework uniquely disaggregates the control problem by transforming it into a multifaceted game via logarithmic barrier functions (BFs): it models humanâmachine cooperation as a positive-sum game oriented toward shared objectives, and disturbance rejection as a zero-sum game tailored for robustness enhancement. To capture the nonideal human decision-making, we integrate the level-$k$reasoning framework to model the operatorâs bounded cognitive dynamics. The corresponding coupled HamiltonâJacobiâIsaacs (HJI) equations for this humanâmachine game are derived, and critically, a rigorous proof of global asymptotic stability (GAS) for the transformed system is provided, establishing a solid theoretical foundation. For online implementation without requiring prior knowledge of the system dynamics, we develop a resource-efficient learning architecture based on the adaptive dynamic programming (ADP) and a novel dynamic event-triggered mechanism (DETM). A key feature of this architecture is a fuzzy logic-based module for dynamic authority allocation, which adaptively adjusts control sharing in real time. Rigorous analysis demonstrates that all signals in the closed-loop system are uniformly ultimately bounded and that Zeno behavior is precluded. Simulation results are presented to validate the effectiveness and superiority of the proposed control strategy.
Various discrete-time zeroing neural network (DTZNN) models have been developed for solving dynamic constrained quadratic programming. However, two challenges persist within the DTZNN framework: first, the theoretical analysis of robustness in disturbance suppression remains insufficient; second, to the best of authors' knowledge, existing DTZNN models have yet to provide a theoretical proof of finite-step convergence. Inspired by the inherent robustness and finite-step convergence of discrete sliding-mode control based on the reaching-law, this article is the first work to integrate reaching-law theory into the DTZNN framework to address the aforementioned challenges, ensuring that the resulting DTZNN exhibits both robustness and finite-step convergence. In addition, a novel hyperbolic type reaching law (HTRL) is designed, which offers advantages in reducing the width of the quasi-sliding-mode region and suppressing chattering. The zeroing neural network (ZNN) based on this HTRL (HTRL-ZNN) is rigorously proven to exhibit effective disturbance suppression robustness and finite-step convergence, with an explicit expression provided for the convergence step length. Finally, the effectiveness and advantages of HTRL-ZNN in solving dynamic constrained quadratic programming are validated through both a numerical example and an application-oriented case.
ABSTRACT This paper deals with the twoâplayer zeroâsum game problem, which is a bounded âgain robust control problem. Finding an analytical solution to the complex HamiltonâJacobiâIssacs (HJI) equation is a challenging task. Hence, a novel Qâlearning algorithm for unknown continuousâtime (CT) affineâinâinputs nonlinear systems is proposed for generating an approximate solution to the HJI equation, which is valid in a local domain due to the use of a local approximator, that is, a Neural Network (NN) structure. The approach is modelâfree and does not require the knowledge of system drift dynamics, and input and disturbance gains. The algorithm learns online from measurements of state variables in real time. To generate the local approximate solution of the HJI equation for the twoâplayer zeroâsum game problem for nonlinear systems, the proposed nonâiterative algorithm requires only a single critic NN instead of the commonly used triple NN approximator structure. A persistence of excitation condition is required to guarantee Uniformly Ultimately Boundedness (UUB) and convergence to the optimal solution. The effectiveness of the proposed Qâlearning approach for the twoâplayer zeroâsum game problem is demonstrated via simulations of a linear Fâ16 aircraft plant and a highly complex nonlinear system. Proof of closedâloop system stability is provided using Lyapunov Analysis, and convergence of the approximate solution to the true saddleâpoint solution is guaranteed in a UUBâsense.
In this paper, we formulate inverse reinforcement learning (IRL) as an expert-learner interaction whereby the optimal performance intent of an expert or target agent is unknown to a learner agent. The learner observes the states and controls of the expert and hence seeks to reconstruct the expert's cost function intent and thus mimics the expert's optimal response. Next, we add non-cooperative disturbances that seek to disrupt the learning and stability of the learner agent. This leads to the formulation of a new interaction we call zero-sum game IRL. We develop a framework to solve the zero-sum game IRL problem that is a modified extension of RL policy iteration (PI) to allow unknown expert performance intentions to be computed and non-cooperative disturbances to be rejected. The framework has two parts: a value function and control action update based on an extension of PI, and a cost function update based on standard inverse optimal control. Then, we eventually develop an off-policy IRL algorithm that does not require knowledge of the expert and learner agent dynamics and performs single-loop learning. Rigorous proofs and analyses are given. Finally, simulation experiments are presented to show the effectiveness of the new approach.
Multi-agent reinforcement learning (MARL) has attracted more and more attention in recent years. It is now widely applied in various fields, including cyber physical systems, smart grid, finance, social network, and among others. The current researches on MARL mainly focus single-time scale, in which the agents have the same decision epoch. While in real applications, it is common that the agents make decisions by different frequencies. In addition, different agents may have separate roles in the system. In this paper, we propose a more general MARL framework by introducing multi-time scale of decision epochs. We assume that agents share information with their neighbors, including state, action, and reward. The global observability of state and action, which is a common assumption, is not required. We propose a decentralized Q-learning algorithm and a modified MADDPG algorithm to solve the problem. The main contributions of this paper are as follows. First, we formulate the multi-time scale multi-agent reinforcement learning (MTMARL) problem. This provides a general framework for the related systems and problems. Second, we provide a networked decentralized multi-time scale multi-agent Q-learning algorithm to solve the problem and prove its convergence. Third, we test the algorithm numerically. The results show that the proposed algorithm performs better than the previous QD-learning and is only slightly worse than the centralized algorithm.
This paper presents a new theory, known as robust dynamic programming, for a class of continuous-time dynamical systems. Different from traditional dynamic programming (DP) methods, this new theory serves as a fundamental tool to analyze the robustness of DP algorithms, and, in particular, to develop novel adaptive optimal control and reinforcement learning methods. In order to demonstrate the potential of this new framework, two illustrative applications in the fields of stochastic and decentralized optimal control are presented. Two numerical examples arising from both finance and engineering industries are also given, along with several possible extensions of the proposed framework.
Sayan Basu Roy, Shubhendu Bhasin, Indra Narayan Kar
Convergence of controller parameters in standard model reference adaptive control (MRAC) requires the system states to be persistently exciting (PE), a restrictive condition to be verified online. A recent data-driven approach, concurrent learning, uses information-rich past data concurrently with the standard parameter update laws to guarantee parameter convergence without the need of the PE condition. This method guarantees exponential convergence of both the tracking and the controller parameter estimation errors to zero, whereas, the classical MRAC merely ensures asymptotic convergence of tracking error to zero. However, the method requires knowledge of the state derivative, at least at the time instances when the state values are stored in memory. The method further assumes knowledge of the control allocation matrix. This paper addresses these limitations by using a memory-based finite-time system identifier in conjunction with a data-driven approach, leading to convergence of both the tracking and the controller parameter estimation errors without the PE condition and knowledge of the system matrices and the state derivative. A Lyapunov based stability proof is included to justify the validity of the proposed data-driven approach. Simulation results demonstrate the efficacy of the suggested method.