Prompt Engineering for LLM-Based Smart Contract Vulnerability Detection: A Systematic Evaluation

Abstract

Smart contract vulnerabilities pose risks to decentralized finance (DeFi) ecosystems, with substantial financial losses from exploits. While large language models (LLMs) offer potential for security auditing, evaluation of prompting strategies and different models for vulnerability detection remains limited. We present a prompt engineering framework comparing seven different strategies (P0-P6) from zero-shot baselines to fine-tuned pipelines. Our prompt designs are implementations from high-performing methodologies: SmartGuard, GPTScan, LLM-SmartAudit, and iAudit. The framework supports evaluation across LLMs on the SmartBugs Curated benchmark with precision, recall, and F1 metrics. We provide: (1) a set of seven prompts (P0-P6) ranging from simple single questions to complex multi-agent and fine-tuned approaches, all producing results in the same JSON format for easy comparison; (2) a testing setup that measures how detection accuracy and API costs change as prompts get more complex; (3) open-source code with tools to run and score each prompt type automatically against any labeled smart contract dataset; and (4) a comparative study showing how each strategy performs on the SmartBugs Curated benchmark.