Privacy-Preserving Record Linkage Over Big Data Platforms

Abstract

Privacy-Preserving Record Linkage (PPRL) integrates sensitive datasets from independent parties without exposing personal identifiers. Although secure multi-party computation (SMC) and homomorphic encryption ensure strong privacy, they suffer from high computational costs and poor scalability. Encoding-based methods, such as Bloom filters, are lightweight but face quality issues at scale owing to saturation and blocking inefficiencies. This study proposes a scalable, modular PPRL framework for distributed platforms. It combines Bloom filter encoding, Hamming-based locality-sensitive hashing (LSH), and Dice similarity within a MapReduce pipeline on a Hadoop distributed file system (HDFS). The system supports decentralized end-to-end linkage under semi-honest or covert adversarial models. Experiments on datasets with$100,000-500,000$records show linear scalability,$7.2 \times$speedup over cryptographic baselines, and recall degradation linked to filter saturation. A regression model captures the execution-candidate volume relationship, thereby aiding system tuning. The framework supports high-throughput, regulation-compliant linkages for healthcare, finance, and public sector use.