Repairing DeFi Vulnerabilities: Benchmarking LLMs with Executable Solidity Exploits

Abstract

Decentralized finance protocols are frequently exploited, creating a demand for fast and reliable repair of vulnerable smart contracts and validation that reflects runtime security. Large language models are an emerging source of patches, yet many evaluations rely on manual checks or self-assessment, which cannot confirm whether attacker profit is actually prevented. We introduce an executable benchmark that replays verified real-world exploits against patched Solidity contracts under a resilient protocol that permits alternate attack paths and controlled state variation. Our framework compiles candidate patches, deploys them on a forked chain, and tests whether the exploit still yields profit. The benchmark covers six test cases drawn from reproducible incidents and is released as open-source. Among the nine evaluated models, GPT-5, GPT-4.1, and Claude Opus 4.1 performed the best, mitigating four of six test cases. Microsoft Phi-4 was the most reliable open-source model, mitigating two of six exploits and producing compilable patches for the remaining cases. No model mitigated the H2O case once resilient checks were enabled, while a simpler access control flaw, BTNFT, was often repaired with minimal edits. Grounding validation in executable exploit replay provides a precise and scalable method to measure whether proposed repairs harden contracts at runtime.