SCOBERT: A Pre-Trained BERT for Smart Contract Vulnerability Detection

Abstract

The introduction of smart contracts has brought new opportunities to the development of blockchain. However, due to the potential security vulnerabilities in smart contracts, accurately identifying these vulnerabilities has become one of the key challenges. Existing detection methods mostly focus on open-source smart contracts and suffer from issues such as low data utilization, insufficient contract information extraction, and high false positive rates. Since the proportion of open-source contracts is extremely low, accurately extracting feature information from non-open-source contracts and identifying vulnerabilities has become an urgent problem to be solved. Therefore, we propose SCOBERT, a pre-training model for blockchain smart contracts. SCOBERT supports downstream tasks related to smart contracts, such as smart contract functionality identification and smart contract vulnerability detection. We develop SCOBERT using Transformer neural architecture and train it through pre-training tasks like MLM(Masked Language Model) and NSP(Next Sentence Prediction). This allows us to make full use of unlabeled smart contract data and capture the semantic information of smart contract opcode sequences effectively. We conduct experiments on three types of contract vulnerabilities: integer overflow, multiplication after division, and reentrancy, and validate them on the validation set. The experimental results demonstrate that the smart contract vulnerability detection method based on the SCOBERT pre-training model performs excellently in terms of accuracy, precision, recall, and F1 score.