Shunhui Ji, Shaoqing Zhu, Pengcheng Zhang, Hai Dong · 5 authors
Smart contracts are commonly deployed for safety-critical applications, the quality assurance of which has been a vital factor. Test cases are standard means to ensure the correctness of data flows in smart contracts. To more efficiently generate test cases with high coverage, we propose an improved genetic algorithm-based test-case generation approach for smart contract data flow testing. Our approach introduces the theory of particle swarm optimization into the genetic algorithm, which reduces the influence brought by the randomness of genetic operations and enhances its capability to find global optima. A set of 30 real smart contracts deployed on Ethereum and GitHub is collected to perform the experimental study, on which our approach is compared with three baseline approaches. The experimental results show that, in most cases, the coverage of the test cases generated by our approach is significantly higher than the baseline approaches with relatively lower numbers of iterations and lower execution time.
Learning smart contract representations can greatly facilitate the development of smart contracts in many tasks such as bug detection and clone detection. Existing approaches for learning program representations are difficult to apply to smart contracts which have insufficient data and significant homogenization. To overcome these challenges, in this paper, we propose SRCL, a novel, self-supervised approach for learning smart contract representations. Unlike existing supervised methods, which are tied on task-specific data labels, SRCL leverages large-scale unlabeled data by self-supervised learning of both local and global information of smart contracts. It automatically extracts structural sequences from abstract syntax trees (ASTs). Then, two discriminators are designed to guide the Transformer encoder to learn local and global semantic features of smart contracts. We evaluate SRCL on a dataset of 75,006 smart contracts collected from Etherscan. Experimental results show that SRCL considerably outperforms the state-of-the-art code representation models on three downstream tasks.
Alex Groce, Kush Jain, Rijnard van Tonder, Goutamkumar Tulajappa Kalburgi · 5 authors
Bitcoin is one of the most prominent distributed software systems in the world. This paper describes an effort to investigate and enhance the effectiveness of the Bitcoin Core fuzzing effort. The effort initially began as a query about how to escape saturation in the fuzzing effort, but developed into a more general exploration. This paper summarizes the outcomes of a two-week focused effort. While the effort found no smoking guns indicating major test/fuzz weaknesses, it produced a large number of additional fuzz corpus entries, increased the set of fuzzers used for Bitcoin Core, and ran mutation analysis of Bitcoin Core fuzz targets, with a comparison to Bitcoin functional tests and other cryptocurrencies’ tests. Our conclusion is that for high quality fuzzing efforts, improvements to the oracle may be the best way to get more out of fuzzing.
Mitchell Olsthoorn, Dimitri Stallenberg, Arie van Deursen, Annibale Panichella
Ethereum is the largest and most prominent smart contract platform. One key property of Ethereum is that once a contract is deployed, it can not be updated anymore. This increases the importance of thoroughly testing the behavior and constraints of the smart contract before deployment. Existing approaches in related work either do not scale or are only focused on finding crashing inputs. In this tool demo, we introduce SynTest-Solidity, an automated test case generation and fuzzing framework for Solidity. SynTest-Solidity implements various metaheuristic search algorithms, including random search (traditional fuzzing) and genetic algorithms (i.e., NSGA-II, MOSA, and DynaMOSA). Finally, we performed a preliminary empirical study to assess the effectiveness of SynTest-Solidity in testing Solidity smart contracts.
Jaeseung Choi, Doyeon Kim, Soomin Kim, Gustavo Grieco · 6 authors
Unlike traditional software, smart contracts have the unique organization in which a sequence of transactions shares persistent states. Unfortunately, such a characteristic makes it difficult for existing fuzzers to find out critical transaction sequences. To tackle this challenge, we employ both static and dynamic analyses for fuzzing smart contracts. First, we statically analyze smart contract bytecodes to predict which transaction sequences will lead to effective testing, and figure out if there is a certain constraint that each transaction should satisfy. Such information is then passed to the fuzzing phase and used to construct an initial seed corpus. During a fuzzing campaign, we perform a lightweight dynamic data-flow analysis to collect data-flow-based feedback to effectively guide fuzzing. We implement our ideas on a practical open-source fuzzer, named SMARTIAN. SMARTIAN can discover bugs in real-world smart contracts without the need for the source code. Our experimental results show that SMARTIAN is more effective than existing state-of-the-art tools in finding known CVEs from real-world contracts. SMARTIAN also outperforms other tools in terms of code coverage.
Ethereum Smart contracts are pieces of code that are run on this blockchain. The correctness of smart contracts is important as they are immutable, their source can be seen by everyone, and they transfer Ether. In this paper, we propose a framework for the automated generation of a set of effective test cases for a given smart contract. We use symbolic execution for generation and mutation testing for selection of test cases. We have evaluated our tool on a set of smart contracts, and our results show how mutation can reduce the size of test suites generated by symbolic execution. Also, by analyzing the survived mutants, we have interesting results about effective test cases that cannot be generated by the symbolic execution engine for smart contracts.
Smart Contracts have noninheritable huge prominence within the recent years. After the Ethereum came into existence in 2015, the execution of smart contract had a great development in blockchain technology. The smart contracts execution faces number of issues to name few reliability, scalability, security. Development of Smart Contracts are not followed by standard software development life cycle. This causes the applications with smart contract(s) cannot perform exhaustive testing as well as it is expensive. The most prominent fields like smart contract testing and analysis of the code for vulnerability attracts many researchers. Targeted on reviewing the techniques and approaches discussed in various selected related papers from IEEE, science direct etc. For each of the selected research work identified open challenges that require further research. Hence this literature review on smart contract testing and analysis of smart contract code intends to emphasize the merits and demerits in smart contracts development process.
Abstract Classifying test executions automatically as pass or fail remains a key challenge in software testing and is referred to as the test oracle problem . It is being attempted to solve this problem with supervised learning over test execution traces. A programme is instrumented to gather execution traces as sequences of method invocations. A small fraction of the programme's execution traces is labelled with pass or fail verdicts. Execution traces are then embedded as fixed length vectors and a neural network (NN) component that uses the line‐by‐line information to classify traces as pass or fail is designed. The classification accuracy of this approach is evaluated using subject programs from different application domains—1. Module from Ethereum Blockchain, 2. Module from PyTorch deep learning framework, 3. Microsoft SEAL encryption library components, 4. Sed stream editor, 5. Nine network protocols from Linux packet identifier, L7‐Filter and 6. Utilities library, commons‐lang for Java. For all subject programs, it was found that test execution classification had high precision, recall and specificity, averaging to 93%, 94% and 96%, respectively, while only training with an average 14% of the total traces. Experiments show that the proposed NN‐based approach is promising in classifying test executions from different application domains.
Jinlei Sun, Song Huang, Changyou Zheng, Tingyong Wang · 6 authors
Integer overflow is a common vulnerability in Ethereum Smart Contracts (ESCs) and often causes huge economic losses. Smart contracts cannot be changed once it is deployed on the blockchain and thus demand further testing. Mutation testing is a fault-based testing method that can effectively improve the sufficiency of a test for smart contracts. However, existing methods cannot efficiently perform mutation testing specifically for integer overflow in ESCs. Therefore, by analyzing integer overflow in ESCs, we propose five special mutation operators to address such vulnerability in terms of detecting sufficiency in ESC testing. An empirical study on 40 open-source ESCs is conducted to evaluate the effectiveness of the proposed mutation operators. Results show that: (1) our proposed mutation operators can reproduce all 179 integer overflow vulnerabilities in 40 smart contracts, and the generated mutants have high compilation pass rate and integer overflow vulnerability generation rate; moreover, (2) the generated mutants can find the shortcomings of existing testing methods for integer overflow vulnerability, thereby providing effective support to improve the sufficiency of the test.
Stefan Driessen, Dario Di Nucci, Geert Monsieur, Damian A. Tamburri · 5 authors
Blockchain and smart contract technology are novel approaches to data and\ncode management that facilitate trusted computing by allowing for development\nin a distributed and decentralized manner. Testing smart contracts comes with\nits own set of challenges which have not yet been fully identified and\nexplored. Although existing tools can identify and discover known\nvulnerabilities and their interactions on the Ethereum blockchain through\nrandom search or symbolic execution, these tools generally do not produce test\nsuites suitable for human oracles. In this paper, we present AGSOLT (Automated\nGenerator of Solidity Test Suites). We demonstrate its efficiency by\nimplementing two search algorithms to automatically generate test suites for\nstand-alone Solidity smart contracts, taking into account some of the\nblockchain-specific challenges. To test AGSOLT, we compared a random search\nalgorithm and a genetic algorithm on a set of 36 real-world smart contracts. We\nfound that AGSOLT is capable of achieving high branch coverage with both\napproaches and even discovered some errors in some of the most popular Solidity\nsmart contracts on Github.\n
Stefan Driessen, Dario Di Nucci, Geert Monsieur, Willem‐Jan van den Heuvel
Blockchain and smart contract technology are novel approaches to data and code management, that facilitate trusted computing by allowing for development in a distributed and decentralized manner. Testing smart contracts comes with its own set of challenges which have not yet been fully identified and explored. Although existing tools can identify and discover known vulnerabilities and their interactions on the Ethereum blockchain through random search or symbolic execution, no framework exists for applying advanced, multi-objective algorithms to create test suites for such smart contracts. In this paper, we present AGSolT (Automated Generator of Solidity Test Suites). We demonstrate its efficiency by implementing two search algorithms to automatically generate test suites for stand-alone Solidity smart contracts, taking into account some of the blockchain-specific challenges. To test AGSolT, we compared a random search algorithm and a genetic algorithm on a set of 36 real-world smart contracts. We found that AGSolT is capable of achieving high branch overage with both approaches and even discovered some errors in some of the most popular Solidity smart contracts on Github.
Smart contract is a computer protocol running on the blockchain, which is widely used in various fields. However, its security problems continue to emerge. Therefore, it is necessary to audit the security of smart contract before it is deployed on the blockchain. Traditional testing methods cannot guarantee a high reliability and correctness required by the smart contract. This paper shows a method for using modeling, simulation and verification language (MSVL) and propositional projection temporal logic (PPTL) to model and verify the smart contract. First, a converter tool SOL2M which can convert Solidity program to MSVL program is developed. Then, the security properties of the smart contract are described by PPTL and a standardized process to verify the contract is designed through UMC4M (Unified Model Checker for MSVL). Finally, an example is given to illustrate the feasibility and practicability of this method in smart contract verification.
Testing is an important technique to improve the quality of Ethereum smart contract programs. However, current work on testing smart contract only focus on static problems of smart contract programs. A data flow oriented test case generation approach for dynamic testing of smart contract programs is still missing. To address this problem, this paper proposes a novel test case generation approach, called ADF-GA (All-uses Data Flow criterion based test case generation using Genetic Algorithm), for Solidity based Ethereum smart contract programs. ADF-GA aims to efficiently generate a valid set of test cases via three stages. First, the corresponding program control flow graph is constructed from the source codes. Second, the generated control flow graph is analyzed to obtain the variable information in the Solidity programs, locate the require statements, and also get the definition-use pairs to be tested. Finally, a genetic algorithm is used to generate test cases, in which an improved fitness function is proposed to calculate the definition-use pairs coverage of each test case with program instrumentation. Experimental studies are performed on several representative Solidity programs. The results show that ADF-GA can effectively generate test cases, achieve better coverage, and reduce the number of iterations in genetic algorithm.
N. Sánchez-Gómez, L. Morales-Trujillo, Javier Gutiérrez, Jesús Torres Valderrama
The use of smart contract augurs a world without intermediaries because the code and the agreements contained therein exist across a distributed, decentralized blockchain network. In software engineering, this collaboration is usually represented by using business process models and smart contracts can be used to implement business collaborations in general and inter-organizational business processes. The validation of this contract and the assurance of its quality are critical for its right application. Early testing in smart contract definition is the fact of this paper. The paper discusses the possibility to use transformation protocols to obtain derived artefacts like test case definitions and smart contract code scaffolds. Generation of derived artefacts significantly reduces the number of defects before deploying the smart contract code in the blockchain network. Transformations protocols are created using model-based software development and modelling techniques. This approach allows to simplify and improve the management and execution of collaborative business processes. This would allow, in addition, the application of systematic mechanisms to evaluate and validate the smart contract and, particularly, the application of early testing techniques which would help to reduce the number of defects and, ultimately, the cost of the final review.
Christof Ferreira Torres, Antonio Ken Iannillo, Arthur Gervais, Radu State
Smart contracts are Turing-complete programs that are executed across a blockchain network. Unlike traditional programs, once deployed they cannot be modified. As smart contracts become more popular and carry more value, they become more of an interesting target for attackers. In recent years, smart contracts suffered major exploits, costing millions of dollars, due to programming errors. As a result, a variety of tools for detecting bugs has been proposed. However, majority of these tools often yield many false positives due to over-approximation or poor code coverage due to complex path constraints. Fuzzing or fuzz testing is a popular and effective software testing technique. However, traditional fuzzers tend to be more effective towards finding shallow bugs and less effective in finding bugs that lie deeper in the execution. In this work, we present CONFUZZIUS, a hybrid fuzzer that combines evolutionary fuzzing with constraint solving in order to execute more code and find more bugs in smart contracts. Evolutionary fuzzing is used to exercise shallow parts of a smart contract, while constraint solving is used to generate inputs which satisfy complex conditions that prevent the evolutionary fuzzing from exploring deeper paths. Moreover, we use data dependency analysis to efficiently generate sequences of transactions, that create specific contract states in which bugs may be hidden. We evaluate the effectiveness of our fuzzing strategy, by comparing CONFUZZIUS with state-of-the-art symbolic execution tools and fuzzers. Our evaluation shows that our hybrid fuzzing approach produces significantly better results than state-of-the-art symbolic execution tools and fuzzers.
Christof Ferreira Torres, Antonio Ken Iannillo, Arthur Gervais, Radu State
<div> <div> <p>Smart contracts are Turing-complete programs that are executed across a blockchain. Unlike traditional programs, once deployed, they cannot be modified. As smart contracts carry more value, they become more of an exciting target for attackers. Over the last years, they suffered from exploits costing millions of dollars due to simple programming mistakes. As a result, a variety of tools for detecting bugs have been proposed. Most of these tools rely on symbolic execution, which may yield false positives due to over-approximation. Recently, many fuzzers have been proposed to detect bugs in smart contracts. However, these tend to be more effective in finding shallow bugs and less effective in finding bugs that lie deep in the execution, therefore achieving low code coverage and many false negatives. An alternative that has proven to achieve good results in traditional programs is hybrid fuzzing, a combination of symbolic execution and fuzzing. In this work, we study hybrid fuzzing on smart contracts and present ConFuzzius, the first hybrid fuzzer for smart contracts. ConFuzzius uses evolutionary fuzzing to exercise shallow parts of a smart contract and constraint solving to generate inputs that satisfy complex conditions that prevent evolutionary fuzzing from exploring deeper parts. Moreover, ConFuzzius leverages dynamic data dependency analysis to efficiently generate sequences of transactions that are more likely to result in contract states in which bugs may be hidden. We evaluate the effectiveness of ConFuzzius by comparing it with state-of-the-art symbolic execution tools and fuzzers for smart contracts. Our evaluation on a curated dataset of 128 contracts and a dataset of 21K real-world contracts shows that our hybrid approach detects more bugs than state-of-the-art tools (up to 23%) and that it outperforms existing tools in terms of code coverage (up to 69%). We also demonstrate that data dependency analysis can boost bug detection up to 18%.</p> </div> </div>
Testing is an important technique to improve the quality of Ethereum smart contract programs. However, current work on testing smart contract only focus on static problems of smart contract programs. A data flow oriented test case generation approach for dynamic testing of smart contract programs is still missing. To address this problem, this paper proposes a novel test case generation approach, called ADF-GA (All-uses Data Flow criterion based test case generation using Genetic Algorithm), for Solidity based Ethereum smart contract programs. ADF-GA aims to efficiently generate a valid set of test cases via three stages. First, the corresponding program control flow graph is constructed from the source codes. Second, the generated control flow graph is analyzed to obtain the variable information in the Solidity programs, locate the require statements, and also get the definition-use pairs to be tested. Finally, a genetic algorithm is used to generate test cases, in which an improved fitness function is proposed to calculate the definition-use pairs coverage of each test case with program instrumentation. Experimental studies are performed on several representative Solidity programs. The results show that ADF-GA can effectively generate test cases, achieve better coverage, and reduce the number of iterations in genetic algorithm.
The challenge of automatically determining the correctness of test executions is referred to as the test oracle problem and is one of the key remaining issues for automated testing. The goal in this paper is to solve the test oracle problem in a way that is general, scalable and accurate. To achieve this, we use supervised learning over test execution traces. We label a small fraction of the execution traces with their verdict of pass or fail. We use the labelled traces to train a neural network (NN) model to learn to distinguish runtime patterns for passing versus failing executions for a given program. Our approach for building this NN model involves the following steps, 1. Instrument the program to record execution traces as sequences of method invocations and global state, 2. Label a small fraction of the execution traces with their verdicts, 3. Designing a NN component that embeds information in execution traces to fixed length vectors, 4. Design a NN model that uses the trace information for classification, 5. Evaluate the inferred classification model on unseen execution traces from the program. We evaluate our approach using case studies from different application domains: 1. Module from Ethereum Blockchain, 2. Module from PyTorch deep learning framework, 3. Microsoft SEAL encryption library components, 4. Sed stream editor, 5. Value pointer library and 6. Nine network protocols from Linux packet identifier, L7-Filter. We found the classification models for all subject programs resulted in high precision, recall and specificity, over 95%, while only training with an average 9% of the total traces. Our experiments show that the proposed neural network model is highly effective as a test oracle and is able to learn runtime patterns to distinguish passing and failing test executions for systems and tests from different application domains.
991012879763303412 HKUST Electronic Theses Coverage-directed differential testing of EVM implementations by Hang Xu thesis 2020 x, 37 pages : illustrations ; 30 cm Ethereum virtual machine(EVM) is the heart of the Ethereum infrastructure and functions as the runtime environment for…Read more ›
With the growing popularity of smart contracts, the problem of validation of their correctness becomes more and more relevant, but at the moment there are no universally recognized tools for their testing. In this paper, we propose to apply the mutational analysis to improve reliability of Solidity smart contracts. We identified wide spread errors in the source code of existing contracts and developed a RegularMutator tool for mutation analysis. It has shown its effectiveness in testing a large smart contracts project. As a result of our analysis, we managed to improve the test suites of the project to find the discovered defects and, increase the quality of the test suite.
Smart contracts are immutable, verifiable, and autonomous pieces of code that can be deployed and ran on blockchain networks like Ethereum. Due to the immutability nature of blockchain, no change is possible on a deployed smart contract or a verified transaction. On the other hand, there are millions of dollars carried by smart contracts in Ethereum blockchain, and hence, a faulty smart contract can lead to a huge monetary loss. Therefore, it is important for smart contract developers to fully test and check the correctness of their code before deploying it on the blockchain. In this paper, we propose a testing mechanism for smart contracts in Solidity language, based on mutation testing. We analyzed a comprehensive list of known bugs in Solidity smart contracts, and designed 10 classes of mutation operators inspired by the real faults. Our experimental results show that our proposed mutation operators can regenerate 10 of 15 famous faulty smart contracts, which have resulted in millions of dollars loss. The results show the effectiveness of our proposed mutation operators in detecting real faults in Solidity smart contracts. We have also extended {\em Universal Mutator } tool with our mutation operators, so that it can automatically generate mutants for smart contracts written in Solidity.