Federated Feature Stores: Real-Time Learning Without Centralized Data Movement

Abstract

The fact that data generated in distributed sources grows exponentially poses great hurdles on the centralized machine learning workflow, especially regarding data privacy, latency and bandwidth overhead. Federated Feature Stores (FFS)Constitute a new paradigm that perpetrates real-time learning in which data is not aggregated. In this paper, one such architecture is provided in which feature engineering, storage and access, are decentralized and co-located with data sources taking advantage of the principles of federated learning. To satisfy the need to maintain data locality but guarantee consistency, low latency inference and privacy regulations, FFS uses a combination of on-device feature computation combined with updating global models. We are working on federated feature synchronization, version control and optimal caching for heterogeneous environments. The experimental analysis over edge clusters and cloud back ends reveals significant bumps in the end-to-end training throughput, inference latency, and privacy preservation over conventional centralized feature pipelines. The given FFS framework provides a scalable, privacy-sensitive, and efficient replacement of older data engineering pipes in the fields of healthcare, finance and the Internet of Things.