Self-healing Dilemmas in Distributed Systems: Fault Correction vs. Fault\n Tolerance

Abstract

Large-scale decentralized systems of autonomous agents interacting via\nasynchronous communication often experience the following self-healing dilemma:\nfault detection inherits network uncertainties making a remote faulty process\nindistinguishable from a slow process. In the case of a slow process without\nfault, fault correction is undesirable as it can trigger new faults that could\nbe prevented with fault tolerance that is a more proactive system maintenance.\nBut in the case of an actual faulty process, fault tolerance alone without\neventually correcting persistent faults can make systems underperforming.\nMeasuring, understanding and resolving such self-healing dilemmas is a timely\nchallenge and critical requirement given the rise of distributed ledgers, edge\ncomputing, the Internet of Things in several energy, transport and health\napplications. This paper contributes a novel and general-purpose modeling of\nfault scenarios during system runtime. They are used to accurately measure and\npredict inconsistencies generated by the undesirable outcomes of fault\ncorrection and fault tolerance as the means to improve self-healing of\nlarge-scale decentralized systems at the design phase. A rigorous experimental\nmethodology is designed that evaluates 696 experimental settings of different\nfault scales, fault profiles and fault detection thresholds in a prototyped\ndecentralized network of 3000 nodes. Almost 9 million measurements of\ninconsistencies were collected in a network, where each node monitors the\nhealth status of another node, while both can defect. The prediction\nperformance of the modeled fault scenarios is validated in a challenging\napplication scenario of decentralized and dynamic in-network data aggregation\nusing real-world data from a Smart Grid pilot project. Findings confirm the\norigin of inconsistencies at design phase.\n