Unveiling the Potential of ChatGPT in Detecting Machine Unauditable Bugs in Smart Contracts: A Preliminary Evaluation and Categorization

Abstract

Smart contracts are becoming an integral part of decentralized applications, yet exploitable bugs in these contracts pose significant threats, often leading to considerable monetary losses. Traditional tools often struggle to identify these bugs, with a recent study indicating that 80% of them are classified as Machine Unauditable Bugs (MUBs), rendering conventional approaches ineffective in addressing such cases. In practice, identifying MUBs requires seasoned expertise and is time-intensive, often stalling project progress. In this work, we present a preliminary evaluation of the performance of ChatGPT, a state-of-the-art large language model, especially in detecting MUBs. Our study first investigates the effectiveness and limitations of ChatGPT in detecting various categories of MUBs with two kinds of prompts, general prompts and guidance prompts, on 246 real-world MUBs collected from Code4rena between 2021 and 2022. Subsequently, we compared the leading tool, SPCON, with ChatGPT on 17 CVE contracts with access control issues (a category of MUBs), and found that ChatGPT exhibited comparable performance but better usability over SPCON. We summarize the implications of our findings for the broader community, shedding light on the model’s capabilities, limitations and potentials in detecting smart contract bugs. Our evaluation dataset and results are released at Github1.