Can Neural Networks Help Smart Contract Testing? An Empirical Study

Abstract

Smart contracts are one of the most successful applications of blockchain technology. In order to guarantee the security of smart contracts, researchers have successively introduced various testing methodologies, including static analysis, symbolic execution, and fuzzing, which contribute to a more rigorous and precise evaluation of smart contract vulnerabilities. Deep learning techniques have been widely applied in traditional software vulnerability detection, while the opposite is true in the field of smart contract testing. Consequently, we anticipate that deep learning can be similarly applied to enhance traditional smart contract vulnerability detection tools. However, there is a lack of empirical study on the performance of deep learning applied to smart contract testing. In order to explore how deep neural networks can help with testing tools on smart contracts, we construct a test framework based on SMARTEST. We manage to train deep learning language models using various neural networks including Transformer, GRU, RNN and test the symbolic execution tool SMARTEST framework with the application of these models on the CVE dataset. Upon analyzing the experimental results, we find that deep neural networks did not surpass traditional language models in enhancing smart contract testing. In terms of accuracy, the SMARTEST tool, which utilizes a statistical 3-gram language model, succeeded in detecting the greatest number of vulnerabilities. Specifically, the 3-gram model was able to identify 69.8% of vulnerabilities in the benchmark set within the first 5 seconds. Based on our experimental findings and thorough analysis, we outline the challenges faced in DNN-assisted smart contract testing and suggest potential directions for improvement.