Marco Stronati, Denis Firsov, Antonio Locascio, Benjamin Livshits
Plonkish is a popular circuit format for developing zero-knowledge proof systems that powers a number of major projects in the blockchain space, responsible for holding billions of dollars and processing millions of transactions per day. These projects, including zero-knowledge rollups, rely on highly hand-optimized circuits whose correctness comes at the cost of time-consuming testing and auditing. In this paper, we present Clap, the first Rust eDSL with a proof system agnostic circuit format, facilitating extensibility, automatic optimizations, and formal assurances for the resultant constraint system. Clap casts the problem of producing Plonkish constraint systems and their witness generators as a semantic-preserving compilation problem. Soundness and completeness of the transformation guarantees the absence of subtle bugs caused by under- or over-constraining. Our experimental evaluation shows that its automatic optimizations achieve better performance compared to manual circuit optimization. The optimizer can also be used to automatically derive custom gates from circuit descriptions.
Multiple sequence alignment (MSA) is a fundamental algorithm in bioinformatics. In a situation when the alignment might need to be protected while revealing the other information such the input sequences and the alignment score, zero knowledge proof can be used. In this paper, a validator checks the consistency between the input sequence and the alignment, and between the alignment and the alignment score. The validator is written in Circom language which will be compile into a circuit. Using a zero knowledge prove system called zkSNARK, a cryptographic proof is generates for the circuit and its input. This proof demonstrates that all inputs are consistent without revealing the actual alignment.
Haochen Sun, J. Li, Change Institutions to: University of Waterloo
The recent surge in artificial intelligence (AI), characterized by the prominence of large language models (LLMs), has ushered in fundamental transformations across the globe. However, alongside these advancements, concerns surrounding the legitimacy of LLMs have grown, posing legal challenges to their extensive applications. Compounding these concerns, the parameters of LLMs are often treated as intellectual property, restricting direct investigations. In this study, we address a fundamental challenge within the realm of AI legislation: the need to establish the authenticity of outputs generated by LLMs. To tackle this issue, we present zkLLM, which stands as the inaugural specialized zero-knowledge proof tailored for LLMs to the best of our knowledge. Addressing the persistent challenge of non-arithmetic operations in deep learning, we introduce tlookup, a parallelized lookup argument designed for non-arithmetic tensor operations in deep learning, offering a solution with no asymptotic overhead. Furthermore, leveraging the foundation of tlookup, we introduce zkAttn, a specialized zero-knowledge proof crafted for the attention mechanism, carefully balancing considerations of running time, memory usage, and accuracy. Empowered by our fully parallelized CUDA implementation, zkLLM emerges as a significant stride towards achieving efficient zero-knowledge verifiable computations over LLMs. Remarkably, for LLMs boasting 13 billion parameters, our approach enables the generation of a correctness proof for the entire inference process in under 15 minutes. The resulting proof, compactly sized at less than 200 kB, is designed to uphold the privacy of the model parameters, ensuring no inadvertent information leakage.
Smart contracts are self-executing programs that can run on a blockchain. Due to the fact of being immutable after their deployment on blockchain, it is crucial to ensure their correctness. For this reason, various approaches for static analysis of smart contracts have been proposed, but they may be on the one hand imprecise or on the other hand difficult to train. In this paper, we propose a novel approach for detecting smart contract vulnerabilities using OpenAI's Generative Pre-trained Transformer 3 (GPT-3) language model. Our approach, called VulntHunt-GPT, uses GPT-3 to examine Ethereum smart contracts in order to identify the most popular vulnerabilities according to OWASP. We train VulntHunt-GPT on a dataset of smart contract functions and vulnerabilities to improve its accuracy. Our experiments show that VulntHunt-GPT outperforms almost all the existing state-of-the-art approaches in detecting a variety of vulnerabilities, including reentrancy attacks, integer overflow, and uninitialized storage. In addition, we conduct a case study to demonstrate the effectiveness of VulntHunt-GPT in detecting real-world smart contract vulnerabilities. We show that VulntHunt-GPT can identify previously unknown vulnerabilities in popular smart contracts, highlighting its potential for improving smart contract security. Our approach provides a promising direction for using natural language processing techniques to improve smart contract security and reduce the risk of smart contract exploits.
We present a Chain-of-Action (CoA) framework for multimodal and retrieval-augmented Question-Answering (QA). Compared to the literature, CoA overcomes two major challenges of current QA applications: (i) unfaithful hallucination that is inconsistent with real-time or domain facts and (ii) weak reasoning performance over compositional information. Our key contribution is a novel reasoning-retrieval mechanism that decomposes a complex question into a reasoning chain via systematic prompting and pre-designed actions. Methodologically, we propose three types of domain-adaptable `Plug-and-Play' actions for retrieving real-time information from heterogeneous sources. We also propose a multi-reference faith score (MRFS) to verify and resolve conflicts in the answers. Empirically, we exploit both public benchmarks and a Web3 case study to demonstrate the capability of CoA over other methods.
Protecting the copyright of large language models (LLMs) has become crucial due to their resource-intensive training and accompanying carefully designed licenses. However, identifying the original base model of an LLM is challenging due to potential parameter alterations. In this study, we introduce HuRef, a human-readable fingerprint for LLMs that uniquely identifies the base model without interfering with training or exposing model parameters to the public. We first observe that the vector direction of LLM parameters remains stable after the model has converged during pretraining, with negligible perturbations through subsequent training steps, including continued pretraining, supervised fine-tuning, and RLHF, which makes it a sufficient condition to identify the base model. The necessity is validated by continuing to train an LLM with an extra term to drive away the model parameters' direction and the model becomes damaged. However, this direction is vulnerable to simple attacks like dimension permutation or matrix rotation, which significantly change it without affecting performance. To address this, leveraging the Transformer structure, we systematically analyze potential attacks and define three invariant terms that identify an LLM's base model. Due to the potential risk of information leakage, we cannot publish invariant terms directly. Instead, we map them to a Gaussian vector using an encoder, then convert it into a natural image using StyleGAN2, and finally publish the image. In our black-box setting, all fingerprinting steps are internally conducted by the LLMs owners. To ensure the published fingerprints are honestly generated, we introduced Zero-Knowledge Proof (ZKP). Experimental results across various LLMs demonstrate the effectiveness of our method. The code is available at https://github.com/LUMIA-Group/HuRef.
The previous smart contract code comment (SCC) generation approaches can be divided into two categories: fine-tuning paradigm-based approaches and information retrieval-based approaches. However, for the fine-tuning paradigm-based approaches, the performance may be limited by the quality of the gathered dataset for the downstream task and they may have knowledge-forgetting issues. While for the information retrieval-based approaches, it is difficult for them to generate high-quality comments if similar code does not exist in the historical repository. Therefore we want to utilize the domain knowledge related to SCC generation in large language models (LLMs) to alleviate the disadvantages of these two types of approaches. In this study, we propose an approach SCCLLM based on LLMs and in-context learning. Specifically, in the demonstration selection phase, SCCLLM retrieves the top-k code snippets from the historical corpus by considering syntax, semantics, and lexical information. In the in-context learning phase, SCCLLM utilizes the retrieved code snippets as demonstrations, which can help to utilize the related knowledge for this task. We select a large corpus from a smart contract community Etherscan.io as our experimental subject. Extensive experimental results show the effectiveness of SCCLLM when compared with baselines in automatic evaluation and human evaluation.
Large Language Models (LLMs), such as ChatGPT, have transformed the field of natural language processing with their capacity for language comprehension and generation of human-like, fluent responses for many downstream tasks. Despite their impressive capabilities, they often fall short in domain-specific and knowledge-intensive domains due to a lack of access to relevant data. Moreover, most state-of-art LLMs lack transparency as they are often accessible only through APIs. Furthermore, their application in critical real-world scenarios is hindered by their proclivity to produce hallucinated information and inability to leverage external knowledge sources. To address these limitations, we propose an innovative system that enhances LLMs by integrating them with an external knowledge management module. The system allows LLMs to utilize data stored in vector databases, providing them with relevant information for their responses. Additionally, it enables them to retrieve information from the Internet, further broadening their knowledge base. The research approach circumvents the need to retrain LLMs, which can be a resource-intensive process. Instead, it focuses on making more efficient use of existing models. Preliminary results indicate that the system holds promise for improving the performance of LLMs in domain-specific and knowledge-intensive tasks. By equipping LLMs with real-time access to external data, it is possible to harness their language generation capabilities more effectively, without the need to continually strive for larger models.
Cryptocurrencies are highly anonymous, poorly regulated in many countries, and can issue tokens at nearzero cost using existing platforms. As a result, there is no shortage of fraudulent cryptocurrencies that raise large sums of money through hype, then disappear and do little actual project development. The prevalence of fraudulent cryptocurrencies not only harms investors but can also prevent sound companies from raising funds. To remedy this situation, it would be useful to develop a method to determine whether a particular cryptocurrency is fraudulent or not. The information in cryptocurrency whitepapers could be useful in detecting fraudulent cryptocurrency, but there are no clear criteria to evaluate the reliability and feasibility of their content. Besides, most studies analyzing whitepapers focus on the success or failure of ICO âfundraisingâ and fail to adequately consider the ongoing development and operation of the project. On the other hand, a few studies have attempted to detect fraudulent cryptocurrencies from whitepapers, but their results suggest the possibility of identifying fraud with high accuracy. The objective of this paper is to build a model to detect fraudulent cryptocurrencies from whitepapers using natural language processing and machine learning techniques, and to verify whether the model has sufficient predictive accuracy in detecting fraud, after solving the problems of previous studies. We collected 250 cryptocurrency whitepapers consisting of 150 frauds and 100 controls, extracted features, and applied multiple machine learning methods to classify frauds and controls. Then analyzed the feature differences between the fraud and control groups, and examined the tendency of fraudulent cryptocurrency whitepapers. We observed 0.841 F1 Score for the best prediction model, which outperforms previous studies. Furthermore, the performance of K-Means, which is unsupervised learning, was not significantly lower than that of other machine learning methods, and a certain level of accuracy was confirmed. Therefore, there is a possibility that K-Means can be used in cases where fraud criteria cannot be clearly defined. We also found that fraudulent cryptocurrency whitepapers used relatively more business and finance-related words. On the other hand, whitepapers in the control group tended to use more blockchain-related technical terms.
As social media has played an essential role in expressing individual ideas, connecting, communicating, and interacting with one another on the digital platform, it has also provided a novel set of data to understand our daily life (Kakulapati, 2021). Non-fungible token (NFT) is an extension of digital token transactions that emerged in 2021. It has also inherited a much more advanced set of data and provides an alternative for information and data communication. As NFT differs from fungible tokens, it is the presentation of the digital and unique transactions of tokens. Based on the blockchain system, NFT is intrinsically the demonstration of decentralised system applied in the transaction of digital work. Each property can be traded freely with the customized value that is set by the NFT creator (Wang et al., 2021). As the value is being set and generated through the smart contract, here the smart contract refers to the adapted contracts that are run by programs that are originally set by individuals, this means that every NFT transaction has complete freedom of setting its own contract conditions, with external interference. NFT transaction is also a transparent process (Wang et al., 2021). While the NFT transaction has been finished, there is a smart contract that has been generated through machines automatically and it is not reversible or editable. The whole transaction process and related information are all documented fully by the machine. Due to the existing transparent framework and the decentralised system, the open data on NFT transactions and predictions are mushrooming ever since 2021.
Manoel Fernando Alonso Gadi, MiguelâĂngel Sicilia
Well labeled natural language corpus data is essential for most natural language processing techniques, especially in specialized fields. However, cohort biases remain a significant challenge in machine learning. The narrow origin of data sampling or human annotators in cohorts is a prevalent issue for machine learning researchers due to its potential to induce bias in the final product. During the development of the CryptoLin corpus for another research project, the authors became concerned about the potential influence of cohort bias on the selection of annotators. Therefore, this paper addresses the question of whether cohort diversity improves the labeling result through the implementation of a repeated annotator process, involving two annotator cohorts and a statistically robust comparison methodology. The utilization of statistical tests, such as the Chi-Square Independence test for absolute frequency tables, and the construction of confidence intervals for Kappa point estimates, facilitates a rigorous analysis of the differences between Kappa estimates. Furthermore, the application of a two-proportion z-test to compare the accuracy scores of UTAD and IE annotators for various pre-trained models, including Vader Sentiment Analysis, TextBlob Sentiment Analysis, Flair NLP library, and FinBERT Financial Sentiment Analysis with BERT, contributes to the advancement of knowledge in this field. The paper utilizes Cryptocurrency Linguo (CryptoLin), a corpus containing 2683 cryptocurrency-related news articles spanning more than three years,and compares two different selection criteria for the annotators. CryptoLin was annotated twice with discrete values representing negative, neutral, and positive news respectively. The first annotation was done by twenty-seven annotators from the same cohort. Each news title was randomly assigned and blindly annotated by three human annotators. The second annotation was carried out by eighty-three annotators from three cohorts. Each news title was randomly assigned and blindly annotated by three human annotators, one in each different cohort. In both annotations, a consensus mechanism using simple voting was applied. The first annotation used the same cohort with students from the same nationality and background. The second used three cohorts with students from a very diverse set of nationalities and educational backgrounds. The results demonstrate that manual labeling done by both groups was acceptable according to inter-rater reliability coefficients Fleissâs Kappa, Krippendorffâs Alpha, and Gwetâs AC1. Preliminary analysis utilizing Vader, Textblob, Flair, and FinBERT confirmed the utility of the data set labeling for further refinement of sentiment analysis algorithms. Our results also highlight that the more diverse annotator pool performed better in all measured aspects.
Android mobile devices are a prime target for a huge number of cyber-criminals as they aim to create malware for disrupting and damaging the servers, clients, or networks. Android malware are in the form of malicious apps, that get downloaded on mobile devices via the Play Store or third-party app markets. Such malicious apps pose serious threats like system damage, information leakage, financial loss to user, etc. Thus, predicting which apps contain malicious behavior will help in preventing malware attacks on mobile devices. Identifying Android malware has become a major challenge because of the ever-increasing number of permissions that applications ask for, to enhance the experience of the users. And most of the times, permissions and other features defined in normal and malicious apps are generally the same. In this paper, we aim to detect Android malware using machine learning, deep learning, and natural language processing techniques. To delve into the problem, we use the Android manifest files which provide us with features like permissions which become the basis for detecting Android malware. We have used the concept of information value for ranking permissions. Further, we have proposed a consensus-based blockchain framework for making more concrete predictions as blockchain have high reliability and low cost. The experimental results demonstrate that the proposed model gives the detection accuracy of 95.44% with the Random Forest classifier. This accuracy is achieved with top 45 permissions ranked according to Information Value.
We focus on a type of linguistic formal reasoning where the goal is to reason\nover explicit knowledge in the form of natural language facts and rules (Clark\net al., 2020). A recent work, named PRover (Saha et al., 2020), performs such\nreasoning by answering a question and also generating a proof graph that\nexplains the answer. However, compositional reasoning is not always unique and\nthere may be multiple ways of reaching the correct answer. Thus, in our work,\nwe address a new and challenging problem of generating multiple proof graphs\nfor reasoning over natural language rule-bases. Each proof provides a different\nrationale for the answer, thereby improving the interpretability of such\nreasoning systems. In order to jointly learn from all proof graphs and exploit\nthe correlations between multiple proofs for a question, we pose this task as a\nset generation problem over structured output spaces where each proof is\nrepresented as a directed graph. We propose two variants of a proof-set\ngeneration model, multiPRover. Our first model, Multilabel-multiPRover,\ngenerates a set of proofs via multi-label classification and implicit\nconditioning between the proofs; while the second model, Iterative-multiPRover,\ngenerates proofs iteratively by explicitly conditioning on the previously\ngenerated proofs. Experiments on multiple synthetic, zero-shot, and\nhuman-paraphrased datasets reveal that both multiPRover models significantly\noutperform PRover on datasets containing multiple gold proofs.\nIterative-multiPRover obtains state-of-the-art proof F1 in zero-shot scenarios\nwhere all examples have single correct proofs. It also generalizes better to\nquestions requiring higher depths of reasoning where multiple proofs are more\nfrequent. Our code and models are publicly available at\nhttps://github.com/swarnaHub/multiPRover\n
Jan 1, 2021·Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
We focus on a type of linguistic formal reasoning where the goal is to reason over explicit knowledge in the form of natural language facts and rules A recent work, named PROVER However, compositional reasoning is not always unique and there may be multiple ways of reaching the correct answer. Thus, in our work, we address a new and challenging problem of generating multiple proof graphs for reasoning over natural language rule-bases. Each proof provides a different rationale for the answer, thereby improving the interpretability of such reasoning systems. In order to jointly learn from all proof graphs and exploit the correlations between multiple proofs for a question, we pose this task as a set generation problem over structured output spaces where each proof is represented as a directed graph. We propose two variants of a proof-set generation model, MULTIPROVER. Our first model, Multilabel-MULTIPROVER, generates a set of proofs via multi-label classification and implicit conditioning between the proofs; while the second model, Iterative-MULTIPROVER, generates proofs iteratively by explicitly conditioning on the previously generated proofs. Experiments on multiple synthetic, zero-shot, and human-paraphrased datasets reveal that both MULTIPROVER models significantly outperform PROVER on datasets containing multiple gold proofs. Iterative-MULTIPROVER obtains state-of-the-art proof F1 in zero-shot scenarios where all examples have single correct proofs. It also generalizes better to questions requiring higher depths of reasoning where multiple proofs are more frequent.
shows that transformers can act as "soft theorem provers" by answering questions over explicitly provided knowledge in natural language. In our work, we take a step closer to emulating formal theorem provers, by proposing PROVER, an interpretable transformer-based model that jointly answers binary questions over rule-bases and generates the corresponding proofs. Our model learns to predict nodes and edges corresponding to proof graphs in an efficient constrained training paradigm. During inference, a valid proof, satisfying a set of global constraints is generated. We conduct experiments on synthetic, hand-authored, and human-paraphrased rule-bases to show promising results for QA and proof generation, with strong generalization performance. First, PROVER generates proofs with an accuracy of 87%, while retaining or improving performance on the QA task, compared to RuleTakers (up to 6% improvement on zero-shot evaluation). Second, when trained on questions requiring lower depths of reasoning, it generalizes significantly better to higher depths (up to 15% improvement). Third, PROVER obtains near perfect QA accuracy of 98% using only 40% of the training data. However, generating proofs for questions requiring higher depths of reasoning becomes challenging, and the accuracy drops to 65% for "depth 5", indicating significant scope for future work.
Mary Maller, Sean Bowe, Markulf Kohlweiss, Sarah Meiklejohn
Ever since their introduction, zero-knowledge proofs have become an important tool for addressing privacy and scalability concerns in a variety of applications. In many systems each client downloads and verifies every new proof, and so proofs must be small and cheap to verify. The most practical schemes require either a trusted setup, as in (pre-processing) zk-SNARKs, or verification complexity that scales linearly with the complexity of the relation, as in Bulletproofs. The structured reference strings required by most zkSNARK schemes can be constructed with multi-party computation protocols, but the resulting parameters are specific to an individual relation. Groth et al. discovered a zk-SNARK protocol with a universal structured reference string that is also updatable, but the string scales quadratically in the size of the supported relations.<br/> <br/>Here we describe a zero-knowledge SNARK, Sonic, which supports a universal and continually updatable structured reference string that scales linearly in size. We also describe a generally useful technique in which untrusted âhelpersâ can compute advice that allows batches of proofs to be verified more efficiently. Sonic proofs are constant size, and in the âhelpedâ batch verification context the marginal cost of verification is comparable with the most efficient SNARKs in the literature<br/>
Representing the semantic relations that exist between two given words (or entities) is an important first step in a wide-range of NLP applications such as analogical reasoning, knowledge base completion and relational information retrieval. A simple, yet surprisingly accurate method for representing a relation between two words is to compute the vector offset (\PairDiff) between the corresponding word embeddings. Despite its empirical success, it remains unclear whether \PairDiff is the best operator for obtaining a relational representation from word embeddings. In this paper, we conduct a theoretical analysis of the \PairDiff operator. In particular, we show that for word embeddings where cross-dimensional correlations are zero, \PairDiff is the only bilinear operator that can minimise the $\ell_{2}$ loss between analogous word-pairs. We experimentally show that for word embedding created using a broad range of methods, the cross-dimensional correlations in word embeddings are approximately zero, demonstrating the general applicability of our theoretical result. Moreover, we empirically verify the implications of the proven theoretical result in a series of experiments where we repeatedly discover \PairDiff as the best bilinear operator for representing semantic relations between words in several benchmark datasets.