Data Leakage and Fair Evaluation in Smart Contract Vulnerability Detection

Abstract

Smart contract vulnerability detection usually uses public datasets for training and evaluation. However, using datasets that contain duplicate and highly similar pairs can bring implicit data leakage between the training and test sets. This may affect the reliability of the evaluation results. To address this issue, this paper constructs a binary classification task related to reentrancy vulnerabilities based on two public datasets, ScrawlD and DIVE. It proposes an overlap-aware evaluation framework for fair evaluation. The framework further identifies data overlap at two distinct tiers: duplicate samples and high-similarity pairs. Two evaluation settings, random split and strict split, are constructed. Experiments are conducted using Logistic Regression, Linear Support Vector Machine (SVM), and Multinomial Naive Bayes (MultinomialNB). Results show that sample overlap exists in both datasets, with a higher degree of overlap in DIVE. And sample leakage between the training and test sets has been eliminated effectively by a strict split. Further analysis reveals that the impact of a strict split on model performance is dataset-dependent. It changes less on ScrawlD but decreases significantly on DIVE. The findings suggest that conventional random splitting tends to inflate performance metrics when sample overlap occurs.