Benchmarking Large Language Models for Ethereum Smart Contract Development

Abstract

The integration of blockchain technology, particularly Ethereum and its smart contract, has revolutionized software programming. Solidity, Ethereum’s main language, is crucial due to its features for blockchain applications. However, the immutable nature of Smart Contract (SC) presents significant security issues, with vulnerabilities leading to financial risks. Meanwhile, Large language models (LLMS ) have transformed software development by enhancing coding efficiency and error detection. Despite their potential, current benchmarks often overlook niche languages like Solidity. This paper introduces the first benchmark to evaluate LLMs in Solidity smart contract generation, aiming to improve automated SC development and blockchain deployment reliability using a Test-Driven Development inspired methodology and pass@k metric. This work not only addresses a significant gap in LLM evaluation for blockchain applications but also extends the capabilities of LLMs in this specialized and critical area of software development.