Evaluating Zero-shot Reasoning with Agentic LLMs for Smart Contract Vulnerability Detection

Abstract

Smart contracts are fundamental to blockchain ecosystems, but remain susceptible to security vulnerabilities that can lead to severe financial losses. Recent advances in agentic AI systems, powered by large language models (LLMs), enable autonomous code analysis and decision-making without explicit task-specific supervision. These systems leverage prompt engineering and zero-shot reasoning to detect vulnerabilities in smart contracts without prior fine-tuning. In this work, we evaluate the effectiveness of agentic LLM-based approaches in identifying vulnerabilities using prompt engineering and zero-shot reasoning across a curated dataset of Solidity smart contracts. Our findings highlight the limitations of current LLMs in automated vulnerability detection, providing insights into their practical applicability for securing decentralized applications. Our best-performing configuration, which integrates zero-shot reasoning with the Tree of Thoughts framework, achieved an F1-score of 73.66%.