LLMs for security analysis for smart contract

Abstract

Smart contracts are immutable once deployed, making security auditing crucial before deployment. Existing automated tools such as Slither are limited to pattern-matching based detection and cannot reason about the contract’s intended behaviour. Manual expert reviews help address this gap but do not scale to the volume of contracts requiring analysis. Large Language Models (LLMs) offer contextual reasoning capabilities that static analysers lack, but single-pass LLM outputs suffer from high false positive rates and unreliable vulnerability detection. This project proposes and evaluates a multi-agent LLM workflow for automated Solidity smart contract vulnerability detection. The system combines Slither static analysis with a staged reasoning pipeline comprising a context agent, a Retrieval Augmented Generation (RAG)-powered research agent, a two-round specialist debate, a judge synthesis agent, and an automated Foundry test generation and verification loop. Slither is retained to cover syntactically identifiable patterns, while the LLM agents focus on contextual and semantic reasoning that static analysis cannot capture. The workflow was evaluated against a labelled dataset of 360 Solidity contracts across twelve vulnerability categories, using a local model (DeepSeek-Coder-V2:16B) and a frontier model (Claude Sonnet 4.6) as baselines. Results show that the workflow substantially improved the local model's detection capability, nearly quintupling recall and doubling F1 over its one-shot baseline, demonstrating that structured multi-agent reasoning can partially compensate for model scale. However, precision remained low across all configurations and certain vulnerability categories remained difficult to detect. The automated test generation stage gives auditors a working starting point rather than requiring tests to be written from scratch. Overall, the results suggest that LLMs, when structured through a multi-agent workflow, show potential as a triage and scaffolding layer in the auditing process, improving coverage and reducing manual effort. However, they are still insufficient to entirely replace expert review.