In the domain of software testing, the generation of test cases is a critical process for detecting system errors and bugs. However, automated test case generation for smart contracts often encounters challenges related to automation, vulnerability diversity, and coverage. This paper presents a novel method, the self-adaptive learning Genetic Algorithm (self-adaptive learning GA), designed to address these issues. Our research methodology incorporates several construction models, namely the Control Dependence Graph (CDG), Control Flow Graph (CFG), and Application Binary Interface (ABI). Initially, the ABI model provides essential information for generating and executing test cases. The CFG model subsequently visualizes potential execution paths through the functions of smart contracts. Ultimately, the CDG model identifies potential vulnerabilities in smart contracts. Using these models, our method enhances automatic test case generation in smart contracts by improving coverage and reducing execution time. We selected a variety of smart contracts from the Decentralized Finance (DeFi) ecosystem for data collection and comparative analysis. The experimental results show superior performance rates, with an average code coverage rate of 98.1%, a total of 3500 vulnerabilities detected, a vulnerability detection rate of 98.7%, a false positive rate of 1.3%, a recall of 98.2%, precision of 98.8%, a path uniqueness rate of 96.4%, false negative rate of 3.5%, an execution time of 25 s, and test case generation time of 16 s. In conclusion, our proposed approach demonstrates a significant improvement over existing methods for test case generation by providing a promising solution for the robustness of smart contracts and security enhancement in the DeFi ecosystem.
The ability to create decentralized applications without the authority of a single entity has attracted numerous developers to build applications using blockchain technology. However, ensuring the correctness of such applications poses significant challenges, as it can result in financial losses or, even worse, a loss of user trust. Testing smart contracts introduces a unique set of challenges due to the additional restrictions and costs imposed by blockchain platforms during test case execution. Therefore, it remains uncertain whether testing techniques developed for traditional software can effectively be adapted to smart contracts. In this study, we propose a multi-objective test selection technique for smart contracts that aims to balance three objectives: time, coverage, and gas usage. We evaluated our approach using a comprehensive selection of real-world smart contracts and compared the results with various test selection methods employed in traditional software systems. Statistical analysis of our experiments, which utilized benchmark Solidity smart contract case studies, demonstrates that our approach significantly reduces the testing cost while still maintaining acceptable fault detection capabilities. This is in comparison to random search, mono-objective search, and the traditional re-testing method that does not employ heuristic search.
Stefan Driessen, Dario Di Nucci, Damian A. Tamburri, W.-J. van den Heuvel
Smart contracts have rapidly gained popularity as self-contained pieces of code, especially those run on the Ethereum blockchain. On the one hand, smart contracts are immutable, have transparent workings, and execute autonomously. On the other hand, these qualities make it essential to properly test the behavior of a smart contract before deploying it. In this paper, we introduce SolAR, a tool and approach for Solidity Automated Test Suite GeneRation. SolAR allows smart contract developers to generate test suites for Solidity smart contracts optimized automatically for branch coverage using either a state-of-the-art genetic algorithm or a fuzzing approach. It enables a novel way to handle blockchain operations—or ChainOps—from a pipeline perspective, entailing a larger-scale as well as more manageable and maintainable service continuity.
Distributed Ledger Technologies are an emerging reality opening the way to new application design paradigms like smart contracts-based distributed applications. If on one side they are creating new markets and opportunities, on the other they are exposing users to new security issues deriving from the scarce maturity in terms of security practices in their design and development. This paper raises a warning about the efficacy of a state-of-the-art software testing tool, namely Mythril, by challenging it with real smart contracts extracted from the Code4arena competitions and comparing its performance with security audits released during the contests. The paper highlights possible root causes of inefficiency, opening the way toward more scalable and efficient smart contract testing tools.
Alexandre Mota, Fei Yang, Cristiano da Silva Teixeira
Nowadays, smart contracts have become increasingly popular and, as with software development in general, testing is the standard method for verifying their correctness. However, smart contracts require a higher level of certainty regarding correctness because they are diffcult to modify once deployed and errors can result in significant financial losses. Therefore, formal verification is essential. In this article, we present our search for a tool capable of formally verifying a real-world smart contract written in a recent version of Solidity.
Hugues Blache, Pierre-Antoine Laharotte, Nour‐Eddin El Faouzi
The deployment of Automated and Connected Vehicles (ACV) into traffic requires certifications and validations guaranteeing high levels of safety, security and reliability. The underlying objective is to gain public acceptance by proving that automation systems might bring out a safer mobility. While plenty of methods to certify these systems are populating the literature, the scenario-based approach stands out by reducing the quantity of required Field tests to validate any new system at stake. In this study, we refine the scenario-based approach by proposing a proof of concept (PoC) for scenario reduction using criticality metrics. For this PoC, we weave a relationship between the a priori criticality of abstract functional scenarios and the words used to generate them. Once, the criticality of a subset of scenarios is qualified based on open field data (HighD), the Latent Dirichlet Allocation (LDA) clustering approach is used to generate topics and feature the relationship between observed criticality and semantics words applied to functional scenarios. The criticality degree of semantics words is used to predict the a priori criticality of unobserved functional scenarios.
Martijn van Meerten, Burcu Kulahcioglu Ozkan, Annibale Panichella
Blockchain systems are prone to concurrency bugs due to the nondeterminism in the delivery order of messages between the distributed nodes. These bugs are hard to detect since they can only be triggered by a specific order or timing of concurrent events in the execution. Systematic concurrency testing techniques, which explore all possible delivery orderings of messages to uncover concurrency bugs, are not scalable to large distributed systems such as blockchains. Random concurrency testing methods search for bugs in a randomly generated set of executions and offer a practical testing method.In this paper, we investigate the effectiveness of random concurrency testing on blockchain systems using a case study on the XRP Ledger of the Ripple blockchain, which maintains one of the most popular cryptocurrencies in the market today. We test the Ripple consensus algorithm of the XRP Ledger by exploring different delivery orderings of consensus protocol messages. Moreover, we design an evolutionary algorithm to guide the random test case generation toward certain system behaviors to discover concurrency bugs more efficiently. Our case study shows that random concurrency testing is effective at detecting concurrency bugs in blockchains, and the evolutionary approach for test generation improves test efficiency. Our experiments could successfully detect the bugs we seeded in the Ripple source code. Moreover, we discovered a previously unknown concurrency bug in the production implementation of Ripple.
Despite the rapid growth of smart contracts, they are suffering numerous security vulnerabilities due to the absence of reliable development and testing. In this article, we apply the metamorphic testing technique to detect smart contract vulnerabilities. Based on the anomalies we observed in vulnerable smart contracts, we define five metamorphic relations to detect abnormal gas consumption and account interaction inconsistency of the target smart contract. Through dynamically executing transactions and checking the final violation of metamorphic relations, we determine whether a smart contract is vulnerable. We evaluate our approach on a benchmark of 67 manually annotated smart contracts. The experimental results show that our approach achieves a higher detection rate (TPR, true positive rate) with a lower misreport rate (FDR, false discovery rate) than the other three state-of-the-art tools. These results further suggest that metamorphic testing is a promising method for detecting smart contract vulnerabilities.
Greybox fuzzers require intermediate programs called fuzz drivers to test smart contract APIs. These fuzz drivers use the semi-random inputs (bytes) generated by fuzzers to prepare suitable inputs required to test APIs. Further, fuzz driver also uses this input to decide sequence in which APIs to be invoked and enables the fuzzer to execute the APIs in that sequence to find the vulnerabilities, if any. Manually writing such complex and intelligent fuzz drivers is laborious, requires deep technical skills, hence can be cumbersome and error prone. In this paper, we propose SmartFuzzDriverGen framework to automatically generate fuzz drivers which invoke smart contract APIs using different strategies: unit-level, sequence-based (random, user-defined), and heuristics based. We evaluate the proposed framework by testing a prototype implementation of it with Golang smart contracts (targeted for Hyperledger Fabric platform) and study the effectiveness of the generated fuzz drivers in terms of code coverage as well as bug finding abilities. We observed that fuzzing of APIs in random sequences performed better than the other methods.
Smart contracts are computer programs running on blockchain platforms to manage large sums of cryptocurrency, govern and carry out transactions of assets between multiple parties. The security of smart contracts has attracted great attention, ever since their adoption in the management of massive cryptocurrency transactions. However, current works mainly rely on the priori security patterns, e.g., reentrancy, delegatecall, gasless send and tainted owner. Although these priori security patterns have been proven effective in finding smart contract vulnerabilities, they are very limited in finding deeper design flaws such as permission bugs, bookkeeping bugs, etc. Moreover, the reliability analysis of smart contracts becomes difficult since most smart contracts have little to no documentation for specifications. Fairness is also an important quality attribute of smart contracts and fairness issues have been proved widely existing in smart contracts such as Ponzi schemes. However, fairness has not yet attracted much research attention. Most security and reliability issues can be detected via conformance checking between software implementation and its specification. As dynamic specification mining has been proven an effective way to learn specifications from software execution traces, in this dissertation, we aim to address the aforementioned challenges by proposing a systematic approach to study the security and reliability of smart contracts based on the learned specifications through dynamic specification mining techniques. In general, contract specifications can be categorized into function-level and contract level specifications. This thesis presents the implementation of InvCon, a dynamic invariant detector for Solidity smart contracts. InvCon analyzes past transac tion histories of a contract to generate likely program invariants, which represent function-level properties that must be maintained during function execution. By combining these invariants with finite event traces from historical transactions, a novel counterexample-guided abstraction refinement algorithm is proposed and implemented in SmCon to mine compact and precise contract-level automata. Additionally, the thesis also presents SPCon that mines the role structures of smart contracts from the past transaction histories for the application of permission bug detection. The thesis demonstrates the applications of program invariants in seman tic bug detection and automated fairness verification. An invariant-based semantic test oracle, implemented in a grey-box fuzzing tool ContraMaster, is developed to enhance vulnerability detection in smart contracts, effectively addressing issues like reentrancy, exception disorder, gasless send, and integer overflow/underflow. For fairness issues, a verification framework FairCon is proposed that treats smart contracts as games and focuses on four fairness properties derived from mechanism design and game theory. Mechanism models are employed for bounded fairness property checking, while program invariants are used to analyze unbounded cases. The thesis also explores the applications of contract-level models in model-based testing and access control bug detection. ModCon is introduced as a tool that allows users to provide test models for smart contracts, specifying state definitions, transition relations, pre/post conditions, invariants, and mapping to the contract code. SPCon leverages the aforementioned role mining to identify permission bugs in smart contracts through conformance testing. The evaluation results demonstrate the high accuracy of permission bug detection, uncovering 11 previously unknown bugs in a well-known smart contract benchmark. Briefly, we have successfully derived various smart contract specifications with dy namic specification mining and studied their applications on security and reliability of smart contracts
Blockchain smart contracts have given rise to a variety of interesting and compelling applications and emerged as a revolutionary force for the Internet. Smart contracts from various fields now hold over one trillion dollars worth of virtual coins, attracting numerous attacks. Quite a few practitioners have devoted themselves to developing tools for detecting bugs in smart contracts. One line of efforts revolve around static analysis techniques, which heavily suffer from high false positive rates. Another line of works concentrate on fuzzing techniques. Unfortunately, current fuzzing approaches for smart contracts tend to conduct fuzzing starting from the initial state of the contract, which expends too much energy revolving around the initial state of the contract and thus is usually unable to unearth bugs triggered by other states. Moreover, most existing methods treat each branch equally, failing to take care of the branches that are rare or more likely to possess bugs. This might lead to resources wasted on normal branches. In this paper, we try to tackle these challenges from three aspects: 1) generating function invocation sequences, we explicitly consider data dependencies between functions to facilitate exploring richer states. We further prolong a function invocation sequence$\mathcal {S}_{1}$by appending a new sequence$\mathcal {S}_{2}$, so that the appended sequence$\mathcal {S}_{2}$can start fuzzing from states that are different from the initial state; 2) we incorporate a branch distance-based measure to evolve test cases iteratively towards a target branch; 3) we engage a branch search algorithm to discover rare and vulnerable branches, and design an energy allocation mechanism to take care of exercising these crucial branches. We implement IR-Fuzz and extensively evaluate it over 12K real-world contracts. Empirical results show that: (i) IR-Fuzz achieves 28% higher branch coverage than state-of-the-art fuzzing approaches, (ii) IR-Fuzz detects more vulnerabilities and increases the average accuracy of vulnerability detection by 7% over current methods, and (iii) IR-Fuzz is fast, generating an average of 350 test cases per second. Our implementation and dataset are released athttps://github.com/Messi-Q/IR-Fuzz, hoping to facilitate future research.
Du Jinhu, Song Huang, Xingya Wang, Changyou Zheng · 5 authors
An Ethereum smart contract is an agreement reached by multiple parties, which is guaranteed by blockchain technology to be executed in accordance with the terms expressed in the form of code. Its security needs are particularly prominent due to a large number of digital assets under management. Testing is an effective way to find flaws that threaten the security of smart contracts. However, current smart contract test case generation methods do not regard the impact of other functions in the smart contract on state variables, resulting in the inaccessibility of the control statements related to state variables and low branch coverage of the function under test. To alleviate this problem, this paper proposes SV-Gen. SV-Gen generates test cases for smart contracts through two steps: static analysis and dynamic search. In the first step, SV-Gen considers the read-write relationship between functions and state variables in the smart contract to generate a function invocation sequence for the function to be tested through a backtracking algorithm on state variables. Then the arguments of transactions to invoke each function in the sequence are generated through regex matching to form the primitive test case. In the second step, the primitive test cases constitute an initial population, and a genetic algorithm undertakes the task of evolving them to high branch coverage. The experimental results on one of the VeriSmart datasets show that SV-Gen can effectively enter the control constraints related to state variables and improve the branch coverage of smart contracts.
Abstract: Software testing is part of a set of activities that ensure high-quality software. It primarily aims at revealing defects that have been inserted into software at various stages of its development. In functional testing, test requirements are derived from software specifications. This paper proposes a functional testing/ evaluation that was performed using the functional specification provided during a Model for Creating a Stable Cryptocurrency Using Fiat Currency for Global Electronic Commerce design and verifies the model against the functional requirements. The functional approach of model effectiveness was used to establish model functions first, and then to build criterion measures to assess how well the objectives were met. The functional requirements of the evaluation model were then transformed into test cases. Furthermore, the test case reports were presented as final findings.
Mitchell Olsthoorn, Arie van Deursen, Annibale Panichella
Transaction-reverting statements are key constructs within Solidity that are extensively used for authority and validity checks. Current state-of-the-art search-based testing and fuzzing approaches do not explicitly handle these statements and therefore can not effectively detect security vulnerabilities. In this paper, we argue that it is critical to directly handle and test these statements to assess that they correctly protect the contracts against invalid requests. To this aim, we propose a new approach that improves the search guidance for these transaction-reverting statements based on interprocedural control dependency analysis, in addition to the traditional coverage criteria. We assess the benefits of our approach by performing an empirical study on 100 smart contracts w.r.t. transaction-reverting statement coverage and vulnerability detection capability. Our results show that the proposed approach can improve the performance of Dy-naMOSA, the state-of-the-art algorithm for test case generation. On average, we improve transaction-reverting statement coverage by 14 % (up to 35 %), line coverage by 8 % (up to 32 %), and vulnerability-detection capability by 17 % (up to 50 %).
A smart contract is a pivotal notion in blockchain technology. Distributed applications contain smart contracts verifying the fulfillment of the conditions, which determine the execution of transactions between the blockchain network nodes. Those software-controlled logical conditions are called verification rules. As the number of conditions increases, the complexity of smart contract testing rapidly grows. This paper aims to propose a smart contract testing pattern that significantly limits the needed number of test cases. For evaluation expression with four verification rules, the pattern usage reduces the number of test cases by 68.75% in relation to the full coverage of logical value combinations. With the increase in the number of logical conditions, not only the number of test cases but also their percentage decreases. Starting from seven verification rules in the evaluation expression, the percentage reduction of test cases exceeds 90%. As a result, the cost of preparing and maintaining test case suites may be substantially cut. It should be emphasized that test execution time can be reduced even by 3 orders of magnitude (from seconds to milliseconds). Such an approach is highly important for regression testing, especially when used in continuous software integration, delivery, and deployment approaches.
Ethereum smart contracts may incur security vulnerabilities. Fuzzing is an industry-standard practice to detect them in improving the dependability of programs. Existing fuzz testing techniques for Ethereum smart contracts are insensitive to whether consecutive seeds of the same function are used for fuzzing the smart contract under test. Nonetheless, smart contracts are often designed to have collaborations among different functions for business activity to complete. We wonder whether this mismatch will make fuzzing techniques less effective than they should be. In this paper, to the best of our knowledge, we present the first work to show that security vulnerability detection can be significantly more effective in smart contract fuzzing if the entry functions of recent past test cases can be distinct. The empirical results show that the performance boost can be as large as 10.4% by simply enabling any test case not invoking the same entry functions as a few recent past test cases. The empirical result also shows that the cost-effectiveness also increases by up to 21.9%.
Mitchell Olsthoorn, Dimitri Stallenberg, Arie van Deursen, Annibale Panichella
Ethereum is the largest and most prominent smart contract platform. One key property of Ethereum is that once a contract is deployed, it can not be updated anymore. This increases the importance of thoroughly testing the behavior and constraints of the smart contract before deployment. Existing approaches in related work either do not scale or are only focused on finding crashing inputs. In this tool demo, we introduce SynTest-Solidity, an automated test case generation and fuzzing framework for Solidity. SynTest-Solidity implements various metaheuristic search algorithms, including random search (traditional fuzzing) and genetic algorithms (i.e., NSGA-II, MOSA, and DynaMOSA). Finally, we performed a preliminary empirical study to assess the effectiveness of SynTest-Solidity in testing Solidity smart contracts.