Abstract: Software testing is part of a set of activities that ensure high-quality software. It primarily aims at revealing defects that have been inserted into software at various stages of its development. In functional testing, test requirements are derived from software specifications. This paper proposes a functional testing/ evaluation that was performed using the functional specification provided during a Model for Creating a Stable Cryptocurrency Using Fiat Currency for Global Electronic Commerce design and verifies the model against the functional requirements. The functional approach of model effectiveness was used to establish model functions first, and then to build criterion measures to assess how well the objectives were met. The functional requirements of the evaluation model were then transformed into test cases. Furthermore, the test case reports were presented as final findings.
A smart contract is a pivotal notion in blockchain technology. Distributed applications contain smart contracts verifying the fulfillment of the conditions, which determine the execution of transactions between the blockchain network nodes. Those software-controlled logical conditions are called verification rules. As the number of conditions increases, the complexity of smart contract testing rapidly grows. This paper aims to propose a smart contract testing pattern that significantly limits the needed number of test cases. For evaluation expression with four verification rules, the pattern usage reduces the number of test cases by 68.75% in relation to the full coverage of logical value combinations. With the increase in the number of logical conditions, not only the number of test cases but also their percentage decreases. Starting from seven verification rules in the evaluation expression, the percentage reduction of test cases exceeds 90%. As a result, the cost of preparing and maintaining test case suites may be substantially cut. It should be emphasized that test execution time can be reduced even by 3 orders of magnitude (from seconds to milliseconds). Such an approach is highly important for regression testing, especially when used in continuous software integration, delivery, and deployment approaches.
Mitchell Olsthoorn, Dimitri Stallenberg, Arie van Deursen, Annibale Panichella
Ethereum is the largest and most prominent smart contract platform. One key property of Ethereum is that once a contract is deployed, it can not be updated anymore. This increases the importance of thoroughly testing the behavior and constraints of the smart contract before deployment. Existing approaches in related work either do not scale or are only focused on finding crashing inputs. In this tool demo, we introduce SynTest-Solidity, an automated test case generation and fuzzing framework for Solidity. SynTest-Solidity implements various metaheuristic search algorithms, including random search (traditional fuzzing) and genetic algorithms (i.e., NSGA-II, MOSA, and DynaMOSA). Finally, we performed a preliminary empirical study to assess the effectiveness of SynTest-Solidity in testing Solidity smart contracts.
Shunhui Ji, Shaoqing Zhu, Pengcheng Zhang, Hai Dong · 5 authors
Smart contracts are commonly deployed for safety-critical applications, the quality assurance of which has been a vital factor. Test cases are standard means to ensure the correctness of data flows in smart contracts. To more efficiently generate test cases with high coverage, we propose an improved genetic algorithm-based test-case generation approach for smart contract data flow testing. Our approach introduces the theory of particle swarm optimization into the genetic algorithm, which reduces the influence brought by the randomness of genetic operations and enhances its capability to find global optima. A set of 30 real smart contracts deployed on Ethereum and GitHub is collected to perform the experimental study, on which our approach is compared with three baseline approaches. The experimental results show that, in most cases, the coverage of the test cases generated by our approach is significantly higher than the baseline approaches with relatively lower numbers of iterations and lower execution time.
Alex Groce, Kush Jain, Rijnard van Tonder, Goutamkumar Tulajappa Kalburgi · 5 authors
Bitcoin is one of the most prominent distributed software systems in the world. This paper describes an effort to investigate and enhance the effectiveness of the Bitcoin Core fuzzing effort. The effort initially began as a query about how to escape saturation in the fuzzing effort, but developed into a more general exploration. This paper summarizes the outcomes of a two-week focused effort. While the effort found no smoking guns indicating major test/fuzz weaknesses, it produced a large number of additional fuzz corpus entries, increased the set of fuzzers used for Bitcoin Core, and ran mutation analysis of Bitcoin Core fuzz targets, with a comparison to Bitcoin functional tests and other cryptocurrencies’ tests. Our conclusion is that for high quality fuzzing efforts, improvements to the oracle may be the best way to get more out of fuzzing.
Mitchell Olsthoorn, Dimitri Stallenberg, Arie van Deursen, Annibale Panichella
Ethereum is the largest and most prominent smart contract platform. One key property of Ethereum is that once a contract is deployed, it can not be updated anymore. This increases the importance of thoroughly testing the behavior and constraints of the smart contract before deployment. Existing approaches in related work either do not scale or are only focused on finding crashing inputs. In this tool demo, we introduce SynTest-Solidity, an automated test case generation and fuzzing framework for Solidity. SynTest-Solidity implements various metaheuristic search algorithms, including random search (traditional fuzzing) and genetic algorithms (i.e., NSGA-II, MOSA, and DynaMOSA). Finally, we performed a preliminary empirical study to assess the effectiveness of SynTest-Solidity in testing Solidity smart contracts.
Abstract Classifying test executions automatically as pass or fail remains a key challenge in software testing and is referred to as the test oracle problem . It is being attempted to solve this problem with supervised learning over test execution traces. A programme is instrumented to gather execution traces as sequences of method invocations. A small fraction of the programme's execution traces is labelled with pass or fail verdicts. Execution traces are then embedded as fixed length vectors and a neural network (NN) component that uses the line‐by‐line information to classify traces as pass or fail is designed. The classification accuracy of this approach is evaluated using subject programs from different application domains—1. Module from Ethereum Blockchain, 2. Module from PyTorch deep learning framework, 3. Microsoft SEAL encryption library components, 4. Sed stream editor, 5. Nine network protocols from Linux packet identifier, L7‐Filter and 6. Utilities library, commons‐lang for Java. For all subject programs, it was found that test execution classification had high precision, recall and specificity, averaging to 93%, 94% and 96%, respectively, while only training with an average 14% of the total traces. Experiments show that the proposed NN‐based approach is promising in classifying test executions from different application domains.
Jinlei Sun, Song Huang, Changyou Zheng, Tingyong Wang · 6 authors
Integer overflow is a common vulnerability in Ethereum Smart Contracts (ESCs) and often causes huge economic losses. Smart contracts cannot be changed once it is deployed on the blockchain and thus demand further testing. Mutation testing is a fault-based testing method that can effectively improve the sufficiency of a test for smart contracts. However, existing methods cannot efficiently perform mutation testing specifically for integer overflow in ESCs. Therefore, by analyzing integer overflow in ESCs, we propose five special mutation operators to address such vulnerability in terms of detecting sufficiency in ESC testing. An empirical study on 40 open-source ESCs is conducted to evaluate the effectiveness of the proposed mutation operators. Results show that: (1) our proposed mutation operators can reproduce all 179 integer overflow vulnerabilities in 40 smart contracts, and the generated mutants have high compilation pass rate and integer overflow vulnerability generation rate; moreover, (2) the generated mutants can find the shortcomings of existing testing methods for integer overflow vulnerability, thereby providing effective support to improve the sufficiency of the test.
Stefan Driessen, Dario Di Nucci, Geert Monsieur, Damian A. Tamburri · 5 authors
Blockchain and smart contract technology are novel approaches to data and\ncode management that facilitate trusted computing by allowing for development\nin a distributed and decentralized manner. Testing smart contracts comes with\nits own set of challenges which have not yet been fully identified and\nexplored. Although existing tools can identify and discover known\nvulnerabilities and their interactions on the Ethereum blockchain through\nrandom search or symbolic execution, these tools generally do not produce test\nsuites suitable for human oracles. In this paper, we present AGSOLT (Automated\nGenerator of Solidity Test Suites). We demonstrate its efficiency by\nimplementing two search algorithms to automatically generate test suites for\nstand-alone Solidity smart contracts, taking into account some of the\nblockchain-specific challenges. To test AGSOLT, we compared a random search\nalgorithm and a genetic algorithm on a set of 36 real-world smart contracts. We\nfound that AGSOLT is capable of achieving high branch coverage with both\napproaches and even discovered some errors in some of the most popular Solidity\nsmart contracts on Github.\n
Stefan Driessen, Dario Di Nucci, Geert Monsieur, Willem‐Jan van den Heuvel
Blockchain and smart contract technology are novel approaches to data and code management, that facilitate trusted computing by allowing for development in a distributed and decentralized manner. Testing smart contracts comes with its own set of challenges which have not yet been fully identified and explored. Although existing tools can identify and discover known vulnerabilities and their interactions on the Ethereum blockchain through random search or symbolic execution, no framework exists for applying advanced, multi-objective algorithms to create test suites for such smart contracts. In this paper, we present AGSolT (Automated Generator of Solidity Test Suites). We demonstrate its efficiency by implementing two search algorithms to automatically generate test suites for stand-alone Solidity smart contracts, taking into account some of the blockchain-specific challenges. To test AGSolT, we compared a random search algorithm and a genetic algorithm on a set of 36 real-world smart contracts. We found that AGSolT is capable of achieving high branch overage with both approaches and even discovered some errors in some of the most popular Solidity smart contracts on Github.
N. Sánchez-Gómez, L. Morales-Trujillo, Javier Gutiérrez, Jesús Torres Valderrama
The use of smart contract augurs a world without intermediaries because the code and the agreements contained therein exist across a distributed, decentralized blockchain network. In software engineering, this collaboration is usually represented by using business process models and smart contracts can be used to implement business collaborations in general and inter-organizational business processes. The validation of this contract and the assurance of its quality are critical for its right application. Early testing in smart contract definition is the fact of this paper. The paper discusses the possibility to use transformation protocols to obtain derived artefacts like test case definitions and smart contract code scaffolds. Generation of derived artefacts significantly reduces the number of defects before deploying the smart contract code in the blockchain network. Transformations protocols are created using model-based software development and modelling techniques. This approach allows to simplify and improve the management and execution of collaborative business processes. This would allow, in addition, the application of systematic mechanisms to evaluate and validate the smart contract and, particularly, the application of early testing techniques which would help to reduce the number of defects and, ultimately, the cost of the final review.
Christof Ferreira Torres, Antonio Ken Iannillo, Arthur Gervais, Radu State
Smart contracts are Turing-complete programs that are executed across a blockchain network. Unlike traditional programs, once deployed they cannot be modified. As smart contracts become more popular and carry more value, they become more of an interesting target for attackers. In recent years, smart contracts suffered major exploits, costing millions of dollars, due to programming errors. As a result, a variety of tools for detecting bugs has been proposed. However, majority of these tools often yield many false positives due to over-approximation or poor code coverage due to complex path constraints. Fuzzing or fuzz testing is a popular and effective software testing technique. However, traditional fuzzers tend to be more effective towards finding shallow bugs and less effective in finding bugs that lie deeper in the execution. In this work, we present CONFUZZIUS, a hybrid fuzzer that combines evolutionary fuzzing with constraint solving in order to execute more code and find more bugs in smart contracts. Evolutionary fuzzing is used to exercise shallow parts of a smart contract, while constraint solving is used to generate inputs which satisfy complex conditions that prevent the evolutionary fuzzing from exploring deeper paths. Moreover, we use data dependency analysis to efficiently generate sequences of transactions, that create specific contract states in which bugs may be hidden. We evaluate the effectiveness of our fuzzing strategy, by comparing CONFUZZIUS with state-of-the-art symbolic execution tools and fuzzers. Our evaluation shows that our hybrid fuzzing approach produces significantly better results than state-of-the-art symbolic execution tools and fuzzers.
Christof Ferreira Torres, Antonio Ken Iannillo, Arthur Gervais, Radu State
<div> <div> <p>Smart contracts are Turing-complete programs that are executed across a blockchain. Unlike traditional programs, once deployed, they cannot be modified. As smart contracts carry more value, they become more of an exciting target for attackers. Over the last years, they suffered from exploits costing millions of dollars due to simple programming mistakes. As a result, a variety of tools for detecting bugs have been proposed. Most of these tools rely on symbolic execution, which may yield false positives due to over-approximation. Recently, many fuzzers have been proposed to detect bugs in smart contracts. However, these tend to be more effective in finding shallow bugs and less effective in finding bugs that lie deep in the execution, therefore achieving low code coverage and many false negatives. An alternative that has proven to achieve good results in traditional programs is hybrid fuzzing, a combination of symbolic execution and fuzzing. In this work, we study hybrid fuzzing on smart contracts and present ConFuzzius, the first hybrid fuzzer for smart contracts. ConFuzzius uses evolutionary fuzzing to exercise shallow parts of a smart contract and constraint solving to generate inputs that satisfy complex conditions that prevent evolutionary fuzzing from exploring deeper parts. Moreover, ConFuzzius leverages dynamic data dependency analysis to efficiently generate sequences of transactions that are more likely to result in contract states in which bugs may be hidden. We evaluate the effectiveness of ConFuzzius by comparing it with state-of-the-art symbolic execution tools and fuzzers for smart contracts. Our evaluation on a curated dataset of 128 contracts and a dataset of 21K real-world contracts shows that our hybrid approach detects more bugs than state-of-the-art tools (up to 23%) and that it outperforms existing tools in terms of code coverage (up to 69%). We also demonstrate that data dependency analysis can boost bug detection up to 18%.</p> </div> </div>
Testing is an important technique to improve the quality of Ethereum smart contract programs. However, current work on testing smart contract only focus on static problems of smart contract programs. A data flow oriented test case generation approach for dynamic testing of smart contract programs is still missing. To address this problem, this paper proposes a novel test case generation approach, called ADF-GA (All-uses Data Flow criterion based test case generation using Genetic Algorithm), for Solidity based Ethereum smart contract programs. ADF-GA aims to efficiently generate a valid set of test cases via three stages. First, the corresponding program control flow graph is constructed from the source codes. Second, the generated control flow graph is analyzed to obtain the variable information in the Solidity programs, locate the require statements, and also get the definition-use pairs to be tested. Finally, a genetic algorithm is used to generate test cases, in which an improved fitness function is proposed to calculate the definition-use pairs coverage of each test case with program instrumentation. Experimental studies are performed on several representative Solidity programs. The results show that ADF-GA can effectively generate test cases, achieve better coverage, and reduce the number of iterations in genetic algorithm.
The challenge of automatically determining the correctness of test executions is referred to as the test oracle problem and is one of the key remaining issues for automated testing. The goal in this paper is to solve the test oracle problem in a way that is general, scalable and accurate. To achieve this, we use supervised learning over test execution traces. We label a small fraction of the execution traces with their verdict of pass or fail. We use the labelled traces to train a neural network (NN) model to learn to distinguish runtime patterns for passing versus failing executions for a given program. Our approach for building this NN model involves the following steps, 1. Instrument the program to record execution traces as sequences of method invocations and global state, 2. Label a small fraction of the execution traces with their verdicts, 3. Designing a NN component that embeds information in execution traces to fixed length vectors, 4. Design a NN model that uses the trace information for classification, 5. Evaluate the inferred classification model on unseen execution traces from the program. We evaluate our approach using case studies from different application domains: 1. Module from Ethereum Blockchain, 2. Module from PyTorch deep learning framework, 3. Microsoft SEAL encryption library components, 4. Sed stream editor, 5. Value pointer library and 6. Nine network protocols from Linux packet identifier, L7-Filter. We found the classification models for all subject programs resulted in high precision, recall and specificity, over 95%, while only training with an average 9% of the total traces. Our experiments show that the proposed neural network model is highly effective as a test oracle and is able to learn runtime patterns to distinguish passing and failing test executions for systems and tests from different application domains.
With the growing popularity of smart contracts, the problem of validation of their correctness becomes more and more relevant, but at the moment there are no universally recognized tools for their testing. In this paper, we propose to apply the mutational analysis to improve reliability of Solidity smart contracts. We identified wide spread errors in the source code of existing contracts and developed a RegularMutator tool for mutation analysis. It has shown its effectiveness in testing a large smart contracts project. As a result of our analysis, we managed to improve the test suites of the project to find the discovered defects and, increase the quality of the test suite.
Smart contracts are immutable, verifiable, and autonomous pieces of code that can be deployed and ran on blockchain networks like Ethereum. Due to the immutability nature of blockchain, no change is possible on a deployed smart contract or a verified transaction. On the other hand, there are millions of dollars carried by smart contracts in Ethereum blockchain, and hence, a faulty smart contract can lead to a huge monetary loss. Therefore, it is important for smart contract developers to fully test and check the correctness of their code before deploying it on the blockchain. In this paper, we propose a testing mechanism for smart contracts in Solidity language, based on mutation testing. We analyzed a comprehensive list of known bugs in Solidity smart contracts, and designed 10 classes of mutation operators inspired by the real faults. Our experimental results show that our proposed mutation operators can regenerate 10 of 15 famous faulty smart contracts, which have resulted in millions of dollars loss. The results show the effectiveness of our proposed mutation operators in detecting real faults in Solidity smart contracts. We have also extended {\em Universal Mutator } tool with our mutation operators, so that it can automatically generate mutants for smart contracts written in Solidity.
The blockchain is a key technology that has been adopted in many application areas to increase security and reliability and to avoid the need for a central trusted authority. One of its essential underlying foundations are smart contracts, which are executable programs for managing data or assets on the blockchain. It is crucial that smart contracts are tested thoroughly due to their immutable nature and since even small bugs can lead to huge monetary losses. However, it is not enough to just test smart contracts, it is also important to ensure the quality and completeness of the tests. Hence, we introduce new smart contract specific mutation operators as well as a novel killing condition that is able to detect a deviation in the gas consumptions, i.e., in the monetary value that is required to perform transactions. Moreover, we establish a baseline for mutation testing of smart contracts by applying our method to a replay test suite and by testing about a thousand contracts.
It is crucial that smart contracts are tested thoroughly due to their immutable nature. Even small bugs in smart contracts can lead to huge monetary losses. However, testing is not enough; it is also important to ensure the quality and completeness of the tests. There are already several approaches that tackle this challenge with mutation testing, but their effectiveness is questionable since they only considered small contract samples. Hence, we evaluate the quality of smart contract mutation testing at scale. We choose the most promising of the existing (smart contract specific) mutation operators, analyse their effectiveness in terms of killability and highlight severe vulnerabilities that can be injected with the mutations. Moreover, we improve the existing mutation methods by introducing a novel killing condition that is able to detect a deviation in the gas consumption, i.e., in the monetary value that is required to perform transactions. This paper has a replication package at https://github.com/pieterhartel/Mutation-at-scale
Smart contract is a special program that manages digital assets on blockchain. It is difficult to recover the loss if users make transactions through buggy smart contracts, which cannot be directly fixed. Hence, it is important to ensure the correctness of smart contracts before deploying them. This paper proposes a systematic framework to mutation testing for smart contracts on Ethereum, which is currently the most popular open blockchain for deploying and running smart contracts. Fifteen novel mutation operators have been designed for Ethereum Smart Contracts (ESC), in terms of keyword, global variable/function, variable unit, and error handling. An empirical study on 26 smart contracts in four Ethereum DApps has been conducted to evaluate the effectiveness of mutation testing. The experimental results show that our approach can outperform the coverage-based approach on defect detection rate (96.01% vs. 55.68%). The ESC mutation operators are effective to reveal real defects and we found 117 out of 729 real bug reports are related to our operators. These show the great potential of using mutation testing for quality assurance of ESC.
Mark Mossberg, Felipe Andrés Manzano, Eric Hennenfent, Alex Groce · 8 authors
An effective way to maximize code coverage in software tests is through dynamic symbolic execution$-$a technique that uses constraint solving to systematically explore a program's state space. We introduce an open-source dynamic symbolic execution framework called Manticore for analyzing binaries and Ethereum smart contracts. Manticore's flexible architecture allows it to support both traditional and exotic execution environments, and its API allows users to customize their analysis. Here, we discuss Manticore's architecture and demonstrate the capabilities we have used to find bugs and verify the correctness of code for our commercial clients.
Patrick L. Chapman, Dianxiang Xu, Lin Deng, Yin Xiong
Blockchain in recent years has exploded in popularity with Ethereum being one of the leading blockchain platforms. Solidity is a widely used scripting language for creating smart contracts in Ethereum applications. Quality assurance in Solidity contracts is of critical importance because bugs or vulnerabilities can lead to a considerable loss of financial assets. However, it is unclear what level of quality assurance is provided in many of these applications. Mutation testing is the process of intentionally injecting faults into a target program and then running the provided test suite against the various injected faults. Mutation testing is used to evaluate the effectiveness of a test suite, measuring the test suite’s capability of covering certain types of faults. This thesis presents Deviant, the first implementation of a mutation testing tool for Solidity smart contracts. Deviant implements mutation operators that cover the unique features of Solidity according to our constructed fault model, in addition to traditional mutation operators that exist for other programming languages. Deviant has been applied to five open-source Solidity projects: MetaCoin [30], MultiSigWallet [31], Alice [29], aragonOS [32], and OpenZeppelin [33]. Experimental results show that the provided test suites result in low mutation scores. These results indicate that the provided tests cannot ensure high-level assurance of code quality. Such evaluation results offer important guidelines for Solidity developers to implement more effective tests in order to deliver trustworthy code and reduce the risk of financial loss.
Automatic test generation typically aims to generate inputs that explore new paths in the program under test in order to find bugs. Existing work has, therefore, focused on guiding the exploration toward program parts that are more likely to contain bugs by using an offline static analysis. In this paper, we introduce a novel technique for targeted greybox fuzzing using an online static analysis that guides the fuzzer toward a set of target locations, for instance, located in recently modified parts of the program. This is achieved by first semantically analyzing each program path that is explored by an input in the fuzzer's test suite. The results of this analysis are then used to control the fuzzer's specialized power schedule, which determines how often to fuzz inputs from the test suite. We implemented our technique by extending a state-of-the-art, industrial fuzzer for Ethereum smart contracts and evaluate its effectiveness on 27 real-world benchmarks. Using an online analysis is particularly suitable for the domain of smart contracts since it does not require any code instrumentation---instrumentation to contracts changes their semantics. Our experiments show that targeted fuzzing significantly outperforms standard greybox fuzzing for reaching 83% of the challenging target locations (up to 14x of median speed-up).
We present Harvey, an industrial greybox fuzzer for smart contracts, which are programs managing accounts on a blockchain. Greybox fuzzing is a lightweight test-generation approach that effectively detects bugs and security vulnerabilities. However, greybox fuzzers randomly mutate program inputs to exercise new paths; this makes it challenging to cover code that is guarded by narrow checks, which are satisfied by no more than a few input values. Moreover, most real-world smart contracts transition through many different states during their lifetime, e.g., for every bid in an auction. To explore these states and thereby detect deep vulnerabilities, a greybox fuzzer would need to generate sequences of contract transactions, e.g., by creating bids from multiple users, while at the same time keeping the search space and test suite tractable. In this experience paper, we explain how Harvey alleviates both challenges with two key fuzzing techniques and distill the main lessons learned. First, Harvey extends standard greybox fuzzing with a method for predicting new inputs that are more likely to cover new paths or reveal vulnerabilities in smart contracts. Second, it fuzzes transaction sequences in a targeted and demand-driven way. We have evaluated our approach on 27 real-world contracts. Our experiments show that the underlying techniques significantly increase Harvey's effectiveness in achieving high coverage and detecting vulnerabilities, in most cases orders-of-magnitude faster; they also reveal new insights about contract code.