Using Mutation Testing To Improve and Minimize Test Suites for Smart Contracts

Abstract

This paper presents a successful industrial case study on the application of mutation testing to evaluate and improve test suites for smart contracts. ERCx is a comprehensive, hand-written test suite and framework for smart contract testing, created by Runtime Verification. Despite its thoroughness, hand-written tests can miss edge cases. To address this, we employed mutation testing, which introduces small, syntactic changes, known as mutants, to the program. Mutants that go undetected by the test suite highlight its potential weaknesses, and by presenting them as testing goals, mutation testing helps developers iteratively improve their test suites. In this study, we used mutation testing to expand the ERCx test suite with five new test cases, including one potential vulnerability identified as critical by the ERCx developers. We also developed a test redundancy metric by analyzing pairwise correlation of test data on mutants; we used this redundancy metric to minimize the test suite by removing redundant tests. Finally, we ran both the full and minimized test suites on 106 real-world, faulty ERC-20 contracts to compare the suites' effectiveness and efficiency. Our findings reveal that although the minimized test suite has systematically lower running times compared to the full suite, it still detected faults in 105 of the 106 real-world tokens, retaining nearly all of the full suite's fault-detection capability.