Performance of Open-Source Large Language Models in Smart Contract Vulnerability Detection

Abstract

Smart contracts are central to decentralized applications, enabling automated execution and eliminating intermediaries. However, once deployed on the blockchain, they become immutable, making them susceptible to significant security vulnerabilities that could lead to financial losses and data breaches. Despite advancements in static analysis and symbolic execution for vulnerability detection, these methods remain limited in dynamic environments and large-scale contracts. Recently, large language models (LLMs) have shown potential in code analysis and vulnerability detection, but given concerns over the security of large models, there is an urgent need for evaluations of open-source LLMs to guide their deployment and application. This study aims to assess the performance of state-of-the-art open-source LLMs in detecting smart contract vulnerabilities. Using the Smartbugs dataset, we evaluate four prominent models-OpenAI’s gpt-oss 20B and 120B, Qwen3 235B, and DeepSeek R1 671B. The study focuses on three main areas: 1) evaluating the effectiveness of these models in detecting various vulnerabilities; 2) comparing them with existing detection tools; and 3) analyzing their limitations and potential for future development. The main contributions of this study include the first comprehensive evaluation of open-source LLMs in smart contract vulnerability detection, a detailed comparison with traditional tools, and the release of open experimental methods and data to foster further research in the field. The code used in this study can be found at https://github.com/zumAndenken/Open-Source-LLMSmartContract.