From Code Synthesis to Security Analysis: Evaluation of Open-Source LLMs on Ethereum Smart Contracts

Abstract

Smart contracts are a fundamental building block of blockchain platforms such as Ethereum, yet their development and auditing require specialized expertise and remain highly error-prone. The immutability of deployed smart contracts significantly amplifies the consequences of coding mistakes and security flaws. Recent advances in Large Language Models (LLMs) have shown promise in automating software development and code analysis tasks; however, the reliability of LLM-generated smart contracts and their effectiveness in vulnerability auditing, particularly for Solidity, remains insufficiently explored. In this paper, we present a systematic and automated evaluation pipeline to comparatively assess the performance of open-source LLMs in two critical phases: (i) smart contract generation from natural language specifications, and (ii) smart contract auditing for vulnerability detection. We benchmark multiple open-source models under consistent experimental settings and analyze their correctness, security awareness, and robustness against insecure outputs. Our findings expose significant performance gaps across models and tasks, revealing strengths and limitations of current open-source LLMs in supporting secure smart contract development. This study provides practical insights for researchers and practitioners seeking to apply LLMs to blockchain programming and security assessment.