A study of full-set validation in evovuln for smart contract vulnerability detection

Abstract

This report studies EvoVuln, a smart contract vulnerability analysis pipeline that represents vulnerability knowledge as a natural-language detection method, refines that knowledge through iterative training, and converts the result into an executable detection plan for downstream contract analysis. The report focuses on the Stage 2 update-acceptance mechanism, which determines whether a candidate knowledge revision should replace the current knowledge state during labelled-data refinement. In the baseline implementation, candidate updates are accepted based only on improvement over the previously misclassified subset. While simple and efficient, this rule may accept revisions that improve local error cases while worsening behaviour on other labelled contracts. To address this limitation, this report investigates a modified acceptance rule that adds full-set validation before acceptance, requiring candidate knowledge to reduce total errors on the full labelled training set. The modified rule was implemented as a localised change at the Stage 2 acceptance point, while the broader pipeline structure was retained. Evaluation was conducted on two selected vulnerability types using matched starting snapshots and repeated full-pipeline runs under both policies. The modified rule consistently accepted fewer candidate updates than the baseline policy, indicating stricter filtering during refinement. It also improved average downstream F1 in both vulnerability types, although the form of improvement differed: for price manipulation, recall increased while precision decreased, whereas for access control, average precision, recall, and F1 all improved. These findings suggest that Stage 2 acceptance is a meaningful design choice in iterative LLM-based vulnerability analysis pipelines. Within the scope of the evaluated cases, adding full-set validation was associated with improved downstream detector performance, although the results should be interpreted cautiously given the limited experimental scope and the pipeline’s sensitivity to LLM-related non-determinism.