Exploring the potential of ChatGPT in detecting logical vulnerabilities in smart contracts

Abstract

With the rapid expansion of blockchain applications, smart contracts are becoming increasingly complex, making the automated detection of contract vulnerabilities more critical than ever. Large language models, due to their advanced code comprehensive ability, are considered to have the potential to undertake the task of automated software vulnerability discovery. Although there have been empirical studies on ChatGPT's automated discovery of contract vulnerabilities, the current empirical research has not addressed how well ChatGPT can detect logical vulnerabilities in smart contracts or whether ChatGPT's detection performance for logical vulnerabilities can be improved. To fill this gap, this study collected and organized seven types of logical vulnerability source codes from 6165 real smart contract audit reports and three datasets, such as Web3Bugs, and used this database to validate ChatGPT's detection capability for logical vulnerabilities. To improve ChatGPT's accuracy in detecting logical vulnerabilities, we fine-tuned ChatGPT with a dataset marked with a specific method, achieving an average accuracy rate of 95% for single vulnerability detection per training session. We improved the original marking method to increase further the number of vulnerabilities that a single model can detect. We used a specific completion marking format, ultimately enabling ChatGPT to detect various logical vulnerabilities. In terms of enhancing model scalability, we found a special training set marking method that allows for the addition of detectable vulnerability types through secondary training.