AI-Driven Real-Time Data Synchronization in Distributed Financial Systems

Abstract

With distributed financial microservices, the DualWrite problem frequently results in data discrepancy between payment gateways and in-house ledgers. Conventional reconciliation schemes are based on high-latency batch reconciliation or hard-coded rules, and cannot identify Soft Drifts, small corruptions in the data (e.g. 3% deviation) that resemble normal variance. The paper suggests a real-time reconciliation model that combines an Apache Kafka streaming high-throughput system and an Unsupervised Isolation Forest anomaly detector. The experimental outcomes have shown that although the application of static rules resulted in a Recall rate of only 52.1% (it does not detect soft drifts), the offered AI model attained 100% Recall in all types of drifts. Moreover, the system had a consistent latency of 3.76 ms which was found to be viable in high-frequency trading settings where low-latency and data integrity are of utmost importance.