Semantics-Based, Automated Preparation of Exploratory Data Analysis for Complex Systems

Abstract

Visual Exploratory Data Analysis (EDA) is a key step in data analysis – however, after decades of research, recommending visualizations and their sequences for a human analyst in an exploration-supporting, efficient and repeatable way is still not a solved problem. However, EDA in the empirical assessment of performance and dependability of complex IT systems has key differences from the general setting: system structure and behavior have at least partial specifications, and the EDA process tends to follow established engineering processes (e.g., for diagnosis). Utilizing these differences, in this paper, we propose a novel, semantically driven approach for rapidly setting up analytic notebooks for the IT performance and dependability EDA of complex systems. An ontology-based knowledge base connects observed and inferred operational data with operational semantics and deployment topology; rule-based inference on the knowledge base creates a model of EDA notebook structure and plot recommendations. The model is automatically translated to notebook code and connected to the input data. We also present an open, end-to-end proof of concept implementation of the approach for the transaction duration analysis of Hyper-ledger Fabric, a complex, cross-organizational distributed ledger platform.