Abhimanyu Nag, Samrat Gupta, Sudipan Sinha, Arka Datta
Decentralized Finance (DeFi) governance models have become increasingly complex due to the involvement of numerous independent agents, each with their own incentives and strategies. To effectively analyze these systems, we propose using Multi Agent Influence Diagrams (MAIDs) as a powerful tool for modeling and studying the strategic interactions within DeFi governance. MAIDs allow for a comprehensive representation of the decision-making processes of various agents, capturing the influence of their actions on one another and on the overall governance outcomes. In this paper, we study a simple governance game that approximates real governance protocols and compute the Nash equilibria using MAIDs. We further outline the structure of a MAID in MakerDAO.
The emergence of blockchain and smart contracts has revolutionized various industries by enabling automated code execution. However, the development of smart contracts, rooted in programming languages, inherits common challenges in traditional software development, notably concerning efficiency, reliability, and security. The application of deep learning techniques holds promise for addressing these challenges. However, a critical issue in applying deep learning to smart contracts is the lack of extensive datasets, as smart contracts have only recently emerged compared to traditional programming languages. To address this problem, we propose a novel approach called Compiler-Guided Generation Networks (CGGNet) for augmenting smart contract datasets. In contrast to existing methods, CGGNet utilizes a compiler as an oracle in generative networks, ensuring the augmentation of a valid smart contract. By incorporating the Montecarlo tree search, CGGNet significantly enhances the diversity and validity of the generated contracts, overcoming the limitations posed by GAN-based models in code augmentations. To the best of our knowledge, this is the first study on code augmentation targeting smart contracts. Our experiments show that millions of unique and valid smart contracts can be augmented from thousands of valid smart contracts, and augmented datasets can mitigate the underfitting problem in practical deep learning applications.
Blockchain is a distributed ledger that was introduced to decentralize monetary systems. However, with time, the applications of blockchain in different realms have been identified. Swarm robotics is a field that combines swarm intelligence and robotics to solve real-world problems that cannot be solved by monolithic robots. Collective decision-making is one of the major behaviors implemented by swarm robotics. This study analyzes existing literature on the applications of blockchain in the collective decision-making scenarios in swarm robotics. Consequently, this study introduces a novel taxonomy to study the different applications effectively. The taxonomy categorizes existing literature into (i) application of blockchain in other areas of swarm robotics, (ii) application of blockchain in continuous collective decision-making scenarios, (iii) application of blockchain in discrete collective decision-making scenarios, (iv) application of blockchain in other discrete collective decision-making scenarios, and (v) application of blockchain in the collective perception scenario. Finally, the limitations of existing work such as excessive resource consumption and violation of swarm robotics principles are discussed.
In the absence of external interference, independent multi-agent systems evolve over time, leading to a transition from an ordered to a chaotic state within the internal dynamics of the scene. To assist in maintaining the stability of dynamic systems, this paper explores the design and composition of the scenario. A category of altruistic agents with selfless attributes is introduced into the multi-agent environment, and the decentralized algorithm I-DDPG is employed to train these altruistic agents to perform tasks with altruistic tendencies. Weakly connected self-organization is used to enable cooperation among altruistic agents in addressing internal issues. The study demonstrates a positive correlation between the overall system stability and individual information entropy, as well as the method of obtaining individual altruistic rewards. To investigate the connection between overall scene stability and self-organization among internal individuals, a traffic scenario is constructed with both human-driven and autonomous vehicles. Autonomous vehicles (AVs) represent a type of altruistic agent, and experiments are conducted to explore the relationship between scene stability and internal self-organization among individuals. The study concludes that using I-DDPG improves the learning efficiency of agents compared to DDPG and demonstrates a higher safety level in stress tests. Post-training, agents utilizing weakly connected self-organization in scenarios involving merging and exiting obtain higher altruistic rewards. This approach aids human-driven vehicles in achieving their intended goals, contributing to an overall improvement in the stability of dynamic systems.
The paper introduces an advanced Decentralized Energy Marketplace (DEM) integrating blockchain technology and artificial intelligence to manage energy exchanges among smart homes with energy storage systems. The proposed framework uses Non-Fungible Tokens (NFTs) to represent unique energy profiles in a transparent and secure trading environment. Leveraging Federated Deep Reinforcement Learning (FDRL), the system promotes collaborative and adaptive energy management strategies, maintaining user privacy. A notable innovation is the use of smart contracts, ensuring high efficiency and integrity in energy transactions. Extensive evaluations demonstrate the system's scalability and the effectiveness of the FDRL method in optimizing energy distribution. This research significantly contributes to developing sophisticated decentralized smart grid infrastructures. Our approach broadens potential blockchain and AI applications in sustainable energy systems and addresses incentive alignment and transparency challenges in traditional energy trading mechanisms. The implementation of this paper is publicly accessible at \url{https://github.com/RasoulNik/DEM}.
Existing permissionless sharded-Blockchains come on the scene. However, there is a lack of systematic formulations and experiments regarding the behaviors of individual miners. In this article, we interpret block mining in a permissionless sharded-Blockchain as a repeated$M$-player noncooperative game with finite actions, and propose a new multiagent deep reinforcement learning (MADRL) framework to allow the miners to maximize their profits in a decentralized fashion by scheduling their resources across the shards without centralized coordination. We formulate the rewards, and design a two-scale action space for each miner to reduce the action space and expedite convergence. We also propose a new MADRL model, named Rainbow-WoLF-PHC, which allows each miner to learn its resource allocation online and converge fast to a mixed strategy Nash equilibrium. Extensive experiments show the superiority of the Rainbow-WoLF-PHC to its alternatives in terms of convergence, stability, and profitable actions. This work provides a prosperous design of an end-user-friendly permissionless sharded-Blockchain.
Most blockchain systems currently adopt resource-consuming protocols to achieve consensus between miners; for example, the Proof-of-Work (PoW) and Practical Byzantine Fault Tolerant (PBFT) schemes, which have a high consumption of computing/communication resources and usually require reliable communications with bounded delay. However, these protocols may be unsuitable for Internet of Things (IoT) networks because the IoT devices are usually lightweight, battery-operated, and deployed in an unreliable wireless environment. Therefore, this paper studies an efficient consensus protocol for blockchain in IoT networks via reinforcement learning. Specifically, the consensus protocol in this work is designed on the basis of the Proof-of-Communication (PoC) scheme directly in a single-hop wireless network with unreliable communications. A distributed MultiAgent Reinforcement Learning (MARL) algorithm is proposed to improve the efficiency and fairness of consensus for miners in the blockchain system. In this algorithm, each agent uses a matrix to depict the efficiency and fairness of the recent consensus and tunes its actions and rewards carefully in an actor-critic framework to seek effective performance. Empirical results from the simulation show that the fairness of consensus in the proposed algorithm is guaranteed, and the efficiency nearly reaches a centralized optimal solution.
In this paper, we formulate inverse reinforcement learning (IRL) as an expert-learner interaction whereby the optimal performance intent of an expert or target agent is unknown to a learner agent. The learner observes the states and controls of the expert and hence seeks to reconstruct the expert's cost function intent and thus mimics the expert's optimal response. Next, we add non-cooperative disturbances that seek to disrupt the learning and stability of the learner agent. This leads to the formulation of a new interaction we call zero-sum game IRL. We develop a framework to solve the zero-sum game IRL problem that is a modified extension of RL policy iteration (PI) to allow unknown expert performance intentions to be computed and non-cooperative disturbances to be rejected. The framework has two parts: a value function and control action update based on an extension of PI, and a cost function update based on standard inverse optimal control. Then, we eventually develop an off-policy IRL algorithm that does not require knowledge of the expert and learner agent dynamics and performs single-loop learning. Rigorous proofs and analyses are given. Finally, simulation experiments are presented to show the effectiveness of the new approach.
Decentralized autonomous organizations (DAOs) have become an indispensable part of digital infrastructure in recent years. The unique organizational characteristics and functional structure empower them to become an effective tool for solving corporate governance issues, including contract risks, principal-agent dilemmas, etc. However, DAOs themselves also face a variety of governance issues. On one hand, as a new economic organization model, the existing corporate governance theories and methods are no longer fully applicable to DAOs. On the other hand, unpredictable logic vulnerabilities and code loopholes in the governance mechanism might cause devastating damage to DAOs. The parallel intelligence theory based on the ACP method (i.e., artificial systems, computational experiments, and parallel execution) is an elegant research paradigm and a practical approach tailored to solving these challenges. As such, we propose a novel parallel governance framework for DAOs based on the parallel intelligence theory and further discuss its technical methodology and implementation model. Furthermore, we construct a parallel governance system for GnosisDAO and conduct computational experiments to validate the effectiveness of its governance mechanism. The experimental results not only confirm the defects of the GnosisDAO governance mechanism but also illustrate parallel governance as a useful research direction to solve existing governance problems of DAOs.
Neuromorphic intelligent systems are motivated by the observation that biological organisms - from algae to primates - excel in swiftly sensing their environment, reacting promptly to its perils and opportunities. Furthermore, biological organisms function more resiliently than our most advanced machines, with a fraction of their power requirements. Taking inspiration from how primates and humans have successfully evolved higher cognitive intelligence within social constructs, this paper proposes neuromorphic systems to be built and governed on a public distributed ledger platform. However, following in the footsteps of generic AI research, neuromorphic benchmarks and algorithms are developed in isolation. Furthermore, as a relatively niche research field, there is limited access to the actual neuromorphic sensors and large publicly available curated data, exacerbating the slow research progress. Nonetheless, centralized neuromorphic datasets and algorithms pose a threat to secure closed-loop behavior and learning outcomes, both commonly modulated in biological organisms via social interactions. This paper makes the case for early adoption of distributed ledger technology by neuromorphic systems and benchmarks to avoid the pitfalls endured by AI research - showcasing competing event-based gesture recognition systems on the Ethereum smart contract platform. This shift towards real-world and dynamic systems on a distributed ledger platform will improve collaboration among neuromorphic researchers while enabling healthy competition via incentives. Smart contract protocols allow model behavior monitoring, setting new learning tasks and increase in baseline performance, and naturally provides a governance framework for evolving neuromorphic systems. The code is publicly made available at: https://ist.github.com/BruceFan123.
A total of 12 years have been passed since this Department was created in 2010 as the first academic forum dedicated to cyber-physical-social systems (CPSS), with the first CPSS research article on the field: “The Emergence of Intelligent Enterprises: From CPS to CPSS.” What has happened and changed during the past decade? A brief reflection and review are presented here with a focus on digital twins in CPS versus parallel intelligence in CPSS, and their relationship to blockchain intelligence, smart contracts, metaverses, DAO, Web3, and decentralized science. The concept of DeMetaverses is thus introduced and interpreted as a DAO-based decentralized autonomous metaverse. The characteristics, mechanism, and impact of DeMetaverses are discussed with a vision for achieving an integrated human, artificial, natural, and organizational intelligence that would transform our world into “6S” societies.
Monte Carlo Algorithm is one of the various algorithms of Reinforcement Learning. It is used with problems that have finite states because of the memory problem. It must remember all the experiences to learn and make a decision. If the agent faces an unseen state, The Agent cannot use the experience for the decision to take the action. With these problems, we proposed the algorithm named The Sharing of Similar Knowledge on Monte Carlo Algorithm (SSKMC) to help Monte Carlo conducted with infinite states and leverage the old experience to decide the action when the agent faces a new experience (unseen state). In this paper, we tested the proposed algorithm with the Cryptocurrency Trading problem (Bitcoin) and compared the testing result of the proposed algorithm to Deep Reinforcement Learning (DRL). By the testing result of The Proposed algorithm makes net worth growth more than the DRL method by 1.55%.
Blockchain is an essentially distributed database recording all transactions or digital events among participating parties. Each transaction in the records is approved and verified by consensus of the participants in the system that requires solving a hard mathematical puzzle, which is known as proof-of-work. To make the approved records immutable, the mathematical puzzle is not trivial to solve and therefore consumes substantial computing resources. However, it is energy-wasteful to have many computational nodes installed in the blockchain competing to approve the records by just solving a meaningless puzzle. Here, we pose proof-of-work as a reinforcement-learning problem by modeling the blockchain growing as a Markov decision process, in which a learning agent makes an optimal decision over the environment's state, whereas a new block is added and verified. Specifically, we design the block verification and consensus mechanism as a deep reinforcement-learning iteration process. As a result, our method utilizes the determination of state transition and the randomness of action selection of a Markov decision process, as well as the computational complexity of a deep neural network, collectively to make the blocks not easy to recompute and to preserve the order of transactions, while the blockchain nodes are exploited to train the same deep neural network with different data samples (state-action pairs) in parallel, allowing the model to experience multiple episodes across computing nodes but at one time. Our method is used to design the next generation of public blockchain networks, which has the potential not only to spare computational resources for industrial applications but also to encourage data sharing and AI model design for common problems.
Blockchains can be either permissioned or permissionless. Existing blockchain consensus algorithms used to create permissioned blockchains to not allow new validators to be dynamically added to the blockchain during runtime. This can be a problem when blockchain is used in swarm robotics, where redundancy is required. Besides, popular algorithms such as proof of work (PoW) that offer redundancy consume a lot of resources, making them unsuitable to run on the powerless robots used in swarm robotics. In addition, such algorithms are also vulnerable to threats such as the 51% attack. This research proposes a novel proof-of-identity (PoI) blockchain consensus algorithm to create a dynamically permissioned blockchain to be used in swarm robotics. This algorithm uses a swarm controller to generate a private-public key pair that is used to authorise validators. Tests carried out showed that the PoI algorithm is immune to the 51% attack threat and can run on low-powered devices such as Raspberry Pi 3. Moreover, the consensus time was also shown to be shorter in comparison to PoW.
The incorporation of robots in the social fabric of our society has taken giant leaps, enabled by advances in artificial intelligence and big data. As these robots become increasingly adept at parsing through enormous datasets and making decisions where humans fall short, a significant challenge lies in the analysis of robot behavior. Capturing interactions between robots, humans and IoT devices in traditional structures such as graphs poses challenges in the storage and analysis of large data sets in dense graphs generated by frequent activities. This paper proposes a framework that uses the blockchain for the storage of robotic interactions, and the use of sheaf theory for analysis of these interactions. Applications of our framework for social robots and swarm robots incorporating imperfect information and irrationality on the blockchain sheaf are proposed. This work shows the application of such a framework for various blockchain applications on the spectrum of human-robot interaction, and identifies key challenges that arise as a result of using the blockchain for robotic applications.
Jianbo Du, Wenjie Cheng, Guangyue Lu, Haotong Cao · 7 authors
When using blockchain in mobile systems, computation intensive mining tasks pose great challenges to the processing capabilities of mobile miner equipment. Mobile edge computing (MEC) is an effective solution to alleviating the problem via task offloading. In the mining process, miners compete for rewards through puzzle solving, where only the miner that first completes the process will be rewarded. Thus, miners may wish to pay higher price and use more communication resources in task offloading and more computation resources in task processing for latency reduction. However, there are risks for the miners not profiting from consuming more resources or paying a higher price, so miners are rational in blockchain systems. In order to maximize the rational total profit of all miners, we use an asynchronous advantage actor-critic (A3C) deep reinforcement learning algorithm to obtain the resource pricing and allocation, considering the stochastic properties of wireless channels, and the prospect theory is employed to strike a good balance between risks and rewards. Numerical results show that our proposed A3C based joint optimization algorithm converges fast and outperforms the baseline algorithms in terms of the total reward.
Multi-agent reinforcement learning (MARL) has attracted more and more attention in recent years. It is now widely applied in various fields, including cyber physical systems, smart grid, finance, social network, and among others. The current researches on MARL mainly focus single-time scale, in which the agents have the same decision epoch. While in real applications, it is common that the agents make decisions by different frequencies. In addition, different agents may have separate roles in the system. In this paper, we propose a more general MARL framework by introducing multi-time scale of decision epochs. We assume that agents share information with their neighbors, including state, action, and reward. The global observability of state and action, which is a common assumption, is not required. We propose a decentralized Q-learning algorithm and a modified MADDPG algorithm to solve the problem. The main contributions of this paper are as follows. First, we formulate the multi-time scale multi-agent reinforcement learning (MTMARL) problem. This provides a general framework for the related systems and problems. Second, we provide a networked decentralized multi-time scale multi-agent Q-learning algorithm to solve the problem and prove its convergence. Third, we test the algorithm numerically. The results show that the proposed algorithm performs better than the previous QD-learning and is only slightly worse than the centralized algorithm.
This study applies reinforcement learning (RL) from the AI machine learning field to derive an optimal Bitcoin-like blockchain mining strategy. A salient feature of the RL learning framework is that an optimal (or near-optimal) strategy can be obtained without knowing the details of the blockchain network model. Previously, the most profitable mining strategy was believed to be honest mining encoded in the default blockchain protocol. It was shown later that it is possible to gain more mining rewards by deviating from honest mining. In particular, the mining problem can be formulated as a Markov Decision Process (MDP) which can be solved to give the optimal mining strategy. However, solving the mining MDP requires knowing the values of various parameters that characterize the blockchain network model. In real blockchain networks, these parameter values are not easy to obtain and may change over time. This hinders the use of the MDP model-based solution. In this study, we employ RL to dynamically learn a mining strategy with performance approaching that of the optimal mining strategy. Since the mining MDP problem has a nonlinear objective function (rather than linear functions of standard MDP problems), we design a new multidimensional RL algorithm to solve the problem. Experimental results indicate that, without knowing the parameter values of the mining MDP model, our multidimensional RL mining algorithm can still achieve optimal performance over time-varying blockchain networks.
The scale of criminal networks (e.g. drug syndicates and terrorist networks) extends globally and poses national security threat to many nations as they also tend to be technologically advance (e.g. Dark Web and Silk Road cryptocurrency). Therefore, it is critical for law enforcement agencies to be equipped with the latest tools in criminal network analysis (CNA) to obtain key hidden links (relationships) within criminal networks to preempt and disrupt criminal network structures and activities. Current hidden or missing link predictive models that are based on Social Network Analysis models rely on ML techniques to improve the performance of the models in terms of predictive accuracy and computing power. Given the improvement in the recent performance of Deep Reinforcement Learning (DRL) techniques which could train ML models through self-generated dataset, DRL can be usefully applied to domains with relatively smaller dataset such as criminal networks. The objective of this study is to assess the comparative performance of a CNA hidden link prediction model developed using DRL techniques against classical ML models such as gradient boosting machine (GBM), random forest (RF) and support vector machine (SVM). The experiment results exhibit an improvement in the performance of the DRL model of about 7.4% over the next best performing classical RF model trained within 1500 iterations. The performance of these link prediction models can be scaled up with the parallel processing capabilities of graphical processing units (GPUs), to significantly improve the speed of training the model and the prediction of hidden links.