In MLaaS, DNN models are kept in a server operated by the service provider and inputs to the DNN models are provided by the clients. Such inputs are used to execute the DNN models and classification results are sent back to the client. In MLaaS, the DNN model owner does not reveal the DNN model parameters to the client. MLaaS there are a few trust problems: (a) The server may not be secure and an attacker may send manipulated classification results to the client. In the case of safety-critical systems using such classification in the decision-making process, an attacker may specifically manipulate the classification result to disrupt the operations of the safety-critical system, (b) The server may intentionally send wrong or random classification results without executing the DNN model to respond to a massive number of classification requests from the clients. In this paper, we investigate the problem of verifying DNN model execution by the service provider in an MLaaS paradigm. A proof of DNN model execution will prove that given an input, the DNN model is executed to generate the classification result by providing sequences of outputs of all functions used in the DNN model. As the service provider in MLaaS does not share the DNN model with the client, we need to verify DNN function outcomes without the knowledge of DNN function parameters. Hence zero-knowledge proof can be used for verifying DNN model execution. In this paper, we use Zero-Knowledge Succinct Non-interactive Arguments of Knowledge (zk-SNARKs) which reduces the size of proof and complexity of proof verification considerably. In particular, we use a quadratic arithmetic program-based zkSNARK for DNN model verification. Our main results in this paper are as follows: (a) We have developed a DNN model execution verification method using a QAP-based zkSNARK. (b) We prove that the verification protocol is correct and privacy-preserving. (c) We analyzed the cost of using such a verification protocol.
Vehicle re-identification (ReID) is a hot topic in intelligent city surveillance. With the development of smart cameras and vehicular edge computing (VEC), numerous media data has opened up new possibilities for enhancing the applications of vehicle ReID. However, traditional vehicle re-identification systems face the following challenges: 1) it is difficult to recognize the identities of the vehicles in various views and similar appearance, 2) the current system is hard to be extended to large-scale of cameras in a low-trust VEC environment. To solve these problems, we propose a Blockchain-based Collaborative Vehicle ReID (BCV-ReID) system in this paper. It contains two core parts including Viewpoint-identity Query Net (VQNet) and VehicleChain (VChain). By utilizing the viewpoint information and local details simultaneously, VQNet can distinguish the vehicle identities in various cross-camera scenes. It employs viewpoint queries and spatial self-attention to learn the inherent correlation of the vehicle parts, enhancing the ability to distinguish vehicles among various viewpoints. Then, we integrate VChain with VQNet to realize a collaborative vehicle ReID system. The ReID task is illustrated from the perspective of blockchain transactions. All transactions are validated by a deeply integrated ReID consensus to counter potential malicious attacks. Experiments show that the proposed method achieves comparable results in three famous ReID datasets, as well as outstanding performance in real applications.
Video Surveillance and Tracking Methods
Advanced Neural Network Applications
Advanced Steganography and Watermarking Techniques
Protecting the copyright of large language models (LLMs) has become crucial due to their resource-intensive training and accompanying carefully designed licenses. However, identifying the original base model of an LLM is challenging due to potential parameter alterations. In this study, we introduce HuRef, a human-readable fingerprint for LLMs that uniquely identifies the base model without interfering with training or exposing model parameters to the public. We first observe that the vector direction of LLM parameters remains stable after the model has converged during pretraining, with negligible perturbations through subsequent training steps, including continued pretraining, supervised fine-tuning, and RLHF, which makes it a sufficient condition to identify the base model. The necessity is validated by continuing to train an LLM with an extra term to drive away the model parameters' direction and the model becomes damaged. However, this direction is vulnerable to simple attacks like dimension permutation or matrix rotation, which significantly change it without affecting performance. To address this, leveraging the Transformer structure, we systematically analyze potential attacks and define three invariant terms that identify an LLM's base model. Due to the potential risk of information leakage, we cannot publish invariant terms directly. Instead, we map them to a Gaussian vector using an encoder, then convert it into a natural image using StyleGAN2, and finally publish the image. In our black-box setting, all fingerprinting steps are internally conducted by the LLMs owners. To ensure the published fingerprints are honestly generated, we introduced Zero-Knowledge Proof (ZKP). Experimental results across various LLMs demonstrate the effectiveness of our method. The code is available at https://github.com/LUMIA-Group/HuRef.
Yuhang Wang, Ying He, F. Richard Yu, Bin Song · 5 authors
The Metaverse offers a highly immersive virtual world where users interact with others and objects in real time through their avatars. The integration of Internet of Things (loT) devices plays a pivotal role in achieving seamless synchronization between these two realms. In this article, we explore the utilization of unmanned aerial vehicles (UAVs) for image capture and data transmission to base stations, which then render and optimize images to construct the Metaverse. Considering the limited resources, when the communication or storage resources of base stations are insufficient, UAV tasks can be offloaded to the Web3 cloud servers for storage and processing, so as to avoid wasting the information collected by UAVs. This continuous interaction among the Web3 cloud servers, UAVs, and base stations enables the real world and Metaverse updating simultaneously. To enhance the effectiveness of our approach, we introduce a quantum collective reinforcement learning method, which empowers UAVs with fast training capabilities. This collective learning enables UAVs to autonomously adapt and swiftly respond to novel scenarios, thereby ensuring seamless integration with new environments. Extensive simulations validate the effectiveness of our proposed method in achieving efficient offloading and enhancing overall system performance within the Metaverse and Web3 ecosystem.
The communities of blockchains and distributed ledgers have been stirred up by the introduction of zero-knowledge proofs (ZKPs). Originally designed to solve privacy issues, ZKPs have now evolved into an effective remedy for scalability concerns and are applied in Zcash (internet money like Bitcoin). To enable ZKPs, Rank-1 Constraint Systems (R1CS) offer a verifier for bi-linear equations. To accurately and efficiently represent R1CS, several language tools like Circom, Noir, and Snarky have been proposed to automate the compilation of advanced programs into R1CS. However, due to the flexible nature of R1CS representation, there can be significant differences in the compiled R1CS forms generated from circuit language programs with the same underlying semantics. To address this issue, this paper uses a data-flow-based R1CS paradigm algorithm, which produces a standardized format for different R1CS instances with identical semantics. By using the normalized R1CS format circuits, the complexity of circuits' verification can be reduced. In addition, this paper presents an R1CS normalization algorithm benchmark, and our experimental evaluation demonstrates the effectiveness and correctness of our methods.
The unmanned aerial vehicle (UAV) equipped with mobile-edge computing (MEC) can act as an air base station to provide computing services for Artificial Intelligence of Things (AIoT) devices in remote areas. However, the computation offloading process poses a risk to users’ privacy due to potential information leaks resulting from interactions between UAVs or migration of data between AIoT devices and UAVs. In this article, we proposed a secure aerial computing network that integrates MEC and blockchain technologies to effectively guarantee privacy and security during computation offloading between AIoT devices and UAVs. Additionally, taking into account task offloading scheduling, radio spectrum resource allocation, and computation resource allocation, a joint optimization problem is formulated to minimize the weighted sum of delay and energy consumption throughout the entire computing process. To tackle this issue, we proposed a block coordinate descent (BCD)-based algorithm to solve the mixed-integer and nonconvex problem. Simulation results demonstrate that the proposed algorithm surpasses other baseline approaches.
Chengzu Dong, Jingwen Zhou, Qi An, Frank Jiang · 7 authors
In recent years, edge-based intelligent UAV delivery systems have attracted significant interest from both the academic and industrial sectors. One key obstacle faced by these smart UAV delivery systems is data privacy, as they rely on vast amounts of data from users and UAVs for training machine learning models for person re-identification (ReID) purposes. To tackle this issue, federated learning (FL) has been extensively adopted as a promising solution since it only involves sharing and updating model parameters with a central server, without transferring raw data. However, traditional FL still suffers from the problem of having a single point of failure. In this study, we present a performance optimization method for federated person re-identification using benchmark analysis in blockchain-powered edge-based smart UAV delivery systems. Our method integrates a decentralized FL mechanism enabled by blockchain, which eliminates the necessity for a central server and stores private data on a decentralized permissioned blockchain, thus preventing a single point of failure. We employ the person ReID application in intelligent UAV delivery systems as a representative example to drive our research and examine privacy concerns. Additionally, we introduce the Federated Re-identification Consensus (FRC) protocol to address the scalability issue of the blockchain in supporting UAV delivery systems. The efficiency of our proposed method is illustrated through experiments on energy efficiency, confirmation time, and throughput. We also explore the effects of the incentive mechanism and analyze the system’s resilience under various security attacks. This study offers valuable insights and potential solutions for addressing data privacy and security challenges in the fast-growing domain of smart UAV delivery systems.
Web3 (also known as Web 3.0) metaverse is a blockchain-driven networked, decentralized, and open virtual world. The key feature of the Web3 metaverse is that the ownership of digital assets is recorded by non-fungible token (NFT) protocol on the blockchain. Thus, users are better encouraged to construct Web3 metaverse due to the ownership of their user-generated content (UGC). However, the existing UGC editors mainly face two challenges: they cannot guarantee the uniqueness of UGC; and they are hard-pressed to find a trade-off between model granularity and 3D modeling difficulty. In this article, we design a novel UGC editor for the Web3 metaverse, named MetaCube, to address these challenges. MetaCube applies an artificial intelligence (AI) method to assist the UGC creation for decreasing the 3D modeling difficulty while maintaining the model granularity. To guarantee the uniqueness of UGC, this article proposes 3D Crypto-dropout, a specially designed dropout that can utilize user information to control the UGC creation process and generate unique fine-grained 3D models. Our experimental results demonstrate that the proposed 3D Crypto-dropout can effectively guarantee the uniqueness of UGC from both numerical and human-centered evaluation. Moreover, the existing challenges and open research topics for the uniqueness of UGC are also profoundly discussed.
Generative Adversarial Networks and Image Synthesis
Despite its popularity, the nature of solar energy is highly uncertain and weather dependent, affecting the business viability and investment of solar energy generation, especially for household users. To stabilize the income from solar energy generation, there have been limited traditional options, such as using energy storage to pool excessive solar energy in off-peak periods or financial derivatives from future markets to hedge energy prices. In this paper, we explore a novel idea of "parametric solar energy insurance", by which solar panel owners can insure their solar energy generation based on a verifiable geographically specific index (surface solar irradiation). Parametric solar energy insurance offers opportunities of financial subsidies for insufficient solar energy generation and amortizes the fluctuations of renewable energy generation geographically. Furthermore, we propose to leverage blockchain and remote sensing (satellite imagery) to provide a publicly verifiable platform for solar energy insurance, which not only automates the underwriting and claims of a solar energy insurance policy, but also improves its accountability and transparency. We utilize the state-of-the-art succinct zero-knowledge proofs (zk-SNARK) to realize privacy-preserving blockchain-based solar energy insurance on real-world permissionless blockchain platform Ethereum.
Real-time and vision-based quality control for industrial processes has drawn great interest from both scientists and practitioners, particularly following the transition to Zero Defect Manufacturing (ZDM) and Industry 4.0. Despite considerable progress, most ZDM approaches focus on the accuracy of the inspection process, often neglecting critical factors for application in the shop floor. On one hand, near real-time methods are needed for early defect detection and containment. On the other hand, data scarcity is an issue causing AI methods to overfit. Another concern is the accountability of AI results, since even if an AI pipeline is successfully deployed, its predictions are not verifiable in the long term. In this work, we explore a real-time solution based on lightweight Deep Residual Networks and Blockchain technology to address these issues. Concretely, we propose a two-phase training strategy to boost the performance of baseline classifiers while maintaining low inference times. The performance of the proposed methodology is presented in two different industrial use cases with strict timing requirements, one concerning battery assembly line and the other antenna manufacturing. We validate the proposed method for defect detection and compare the results with common training strategies demonstrating an improvement of 3% and 10% in F1-score and accuracy on the two cases respectively, while lowering inference time by 2.2× compared to existing light architectures. Contributing to the accountability of AI results, we present an IoT framework using Blockchain deployed in Private Ethereum.
Brain tumors are nonlinear and present with variations in their size, form, and textural variation; this might make it difficult to diagnose them and perform surgical excision using magnetic resonance imaging (MRI) scans. The procedures that are currently available are conducted by radiologists, brain surgeons, and clinical specialists. Studying brain MRIs is laborious, error-prone, and time-consuming, but they nonetheless show high positional accuracy in the case of brain cells. The proposed convolutional neural network model, an existing blockchain-based method, is used to secure the network for the precise prediction of brain tumors, such as pituitary tumors, meningioma tumors, and glioma tumors. MRI scans of the brain are first put into pre-trained deep models after being normalized in a fixed dimension. These structures are altered at each layer, increasing their security and safety. To guard against potential layer deletions, modification attacks, and tempering, each layer has an additional block that stores specific information. Multiple blocks are used to store information, including blocks related to each layer, cloud ledger blocks kept in cloud storage, and ledger blocks connected to the network. Later, the features are retrieved, merged, and optimized utilizing a Genetic Algorithm and have attained a competitive performance compared with the state-of-the-art (SOTA) methods using different ML classifiers.
License Plate Recognition (LPR) is essential for the Internet of Vehicles (IoV) since license plates are a necessary characteristic for distinguishing vehicles for traffic management. As the number of vehicles on the road continues to grow, managing and controlling traffic has become increasingly complex. Large cities in particular face significant challenges, including concerns around privacy and the consumption of resources. To address these issues, the development of automatic LPR technology within the IoV has emerged as a critical area of research. By detecting and recognizing license plates on roadways, LPR can significantly enhance management and control of the transportation system. However, implementing LPR within automated transportation systems requires careful consideration of privacy and trust issues, particularly in relation to the collection and use of sensitive data. This study recommends a blockchain-based approach for IoV privacy security that makes use of LPR. A system handles the registration of a user's license plate directly on the blockchain, avoiding the gateway. The database controller may crash as the number of vehicles in the system rises. This paper proposes a privacy protection system for the IoV using license plate recognition based on blockchain. When a license plate is captured by the LPR system, the captured image is sent to the gateway responsible for managing all communications. When the user requires the license plate, the registration is done by a system connected directly to the blockchain, without going through the gateway. Moreover, in the traditional IoV system, the central authority has full authority to manage the binding of vehicle identity and public key. As the number of vehicles increases in the system, it may cause the central server to crash. Key revocation is the process in which the blockchain system analyses the behaviour of vehicles to judge malicious users and revoke their public keys.
As an emerging digital concept offering interconnections across multiple platforms, the metaverse provides digital transformations for various aspects of the physical world, facilitated by a few novel technologies, for example, cloud computing offers data support for the digital world. Humans immersed in the metaverse are digital entities who communicate with others or objects, such that ubiquitous access controls (UACs) are indispensable sectors for multiple platforms. However, in the metaverse, UACs have opened a wide scope of bridges for individuals to shuttle the virtual world, which implies that numerous threats exist at the access layer due to a great pool of entries. In this paper, to solve security issues in the UAC setting of the metaverse, we propose a novel blockchain-based multisignature lock for UAC (BMSL-UAC) scheme. All data institutions reconstruct a consortium blockchain system. In addition, our proposed scheme ensures that only authorized users can access an institution’s data. Finally, we abstract the user’s data access behaviors into the transaction information of the consortium blockchain system to realize full life-cycle data management and traceability. To verify the performance of our scheme, a series of experiments are carried out on the Hyperledger, and evaluation results have demonstrated that the resource consumption, delay, and throughput of this scheme are all within a reasonable range.
On the Internet of Battlefield Things (IoBT), unmanned aerial vehicles (UAVs) provide significant operational advantages. However, the exploitation of the UAV by an untrustworthy entity might lead to security violations or possibly the destruction of crucial IoBT network functionality. The IoBT system has substantial issues related to data tampering and fabrication through illegal access. This paper proposes the use of an intelligent architecture called IoBT-Net, which is built on a convolution neural network (CNN) and connected with blockchain technology, to identify and trace illicit UAV in the IoBT system. Data storage on the blockchain ledger is protected from unauthorized access, data tampering, and invasions. Conveniently, this paper presents a low complexity and robustly performed CNN called LRCANet to estimate AOA for object localization. The proposed LRCANet is efficiently designed with two core modules, called GFPU and stacks, which are cleverly organized with regular and point convolution layers, a max pool layer, and a ReLU layer associated with residual connectivity. Furthermore, the effectiveness of LRCANET is evaluated by various network and array configurations, RMSE, and compared with the accuracy and complexity of the existing state-of-the-art. Additionally, the implementation of tailored drone-based consensus is evaluated in terms of three major classes and compared with the other existing consensus.
Daniel Kang, Tatsunori Hashimoto, Ion Stoica, Yi Sun
As ML models have increased in capabilities and accuracy, so has the complexity of their deployments. Increasingly, ML model consumers are turning to service providers to serve the ML models in the ML-as-a-service (MLaaS) paradigm. As MLaaS proliferates, a critical requirement emerges: how can model consumers verify that the correct predictions were served, in the face of malicious, lazy, or buggy service providers? In this work, we present the first practical ImageNet-scale method to verify ML model inference non-interactively, i.e., after the inference has been done. To do so, we leverage recent developments in ZK-SNARKs (zero-knowledge succinct non-interactive argument of knowledge), a form of zero-knowledge proofs. ZK-SNARKs allows us to verify ML model execution non-interactively and with only standard cryptographic hardness assumptions. In particular, we provide the first ZK-SNARK proof of valid inference for a full resolution ImageNet model, achieving 79\% top-5 accuracy. We further use these ZK-SNARKs to design protocols to verify ML model execution in a variety of scenarios, including for verifying MLaaS predictions, verifying MLaaS model accuracy, and using ML models for trustless retrieval. Together, our results show that ZK-SNARKs have the promise to make verified ML model inference practical.
Tao Li, Yaozheng Fang, Ye Lu, Jinni Yang · 7 authors
Blockchain-based artificial intelligence (BC-AI) has been applied for protecting deep neural network (DNN) data from being tampered with, which is expected to further boost trusted distributed AI applications in many fields. However, due to smart contract execution environment architectural defects, it is challenging for previous BC-AI systems to support computing-intensive tasks on-chain performing such as DNN convolution operations. They have to offload computations and a large amount of data from blockchain to off-chain platforms to execute smart contracts as native code. This failure to take advantage of data locality has become one of the major critical performance bottlenecks in BC-AI system. To this end, in this article, we propose SmartVM with optimization methods to support on-chain DNN inference for BC-AI system. The key idea is to design and optimize the computing mechanism and storage structure of smart contract execution environment according to the characteristics of DNN such as high computational parallelism and large data volume. We decompose SmartVM into three components: 1) a compact DNN-oriented instruction set to describe computations in a short number of instructions to reduce interpretation time. 2) a memory management mechanism to make SmartVM memory dynamic free/allocated according to the size of DNN feature maps. 3) a block-based weight prefetching and parallel computing method to organize each layer's computing and weights prefetching in a pipelined manner. We perform the typical image classification in a private Ethereum blockchain testbed to evaluate SmartVM performance. Experimental results highlight that SmartVM can support DNN inference on-chain with roughly the same efficiency against the native code execution. Compared with the traditional off-chain computing, SmartVM can speed up the overall execution by70×,16×,11×, and12×over LeNet5, AlexNet, ResNet18, and MobileNet, respectively. The memory footprint can be reduced by84%,90.8%,94.3%, and93.7%over the above four models, while offering the same level model accuracy. This article sheds light on the design space of the smart contract virtual machine for DNN computation and is promising to further boost BC-AI applications.
Muhammad Zawish, Nouman Ashraf, Rafay Iqbal Ansari, Steven Davy · 7 authors
6G envisions artificial intelligence (AI) powered solutions for enhancing the quality of service (QoS) in the network and to ensure optimal utilization of resources. In this work, we propose an architecture based on the combination of unmanned aerial vehicles (UAVs), AI, and blockchain for agricultural supply chain management with the purpose of ensuring traceability and transparency, tracking inventories, and contracts. We propose a solution to facilitate on-device AI by generating a roadmap of models with various resource-accuracy trade-offs. A fully convolutional neural network (FCN) model is used for biomass estimation through images captured by the UAV. Instead of a single compressed FCN model for deployment on UAVs, we motivate the idea of iterative pruning to provide multiple task-specific models with various complexities and accuracy. To alleviate the impact of flight failure in a 6G-enabled dynamic UAV network, the proposed model selection strategy will assist UAVs to update the model based on the runtime resource requirements.
Md Masuduzzaman, Tariq Rahim, Anik Islam, Soo Young Shin
This article presents a deep learning (DL) model integrated automated and secure garbage management scheme using unmanned any vehicle (UxV) to minimize the human effort in terms of the traditional garbage management system. Different kinds of UxV (unmanned aerial vehicles, automated guided vehicles, unmanned surface vehicles, unmanned underwater vehicles, etc.) are utilized to establish an automated garbage management scheme to collect and place the garbage both from the ground and sea surfaces. However, due to the limited battery capacity and inadequate resources of different UxV, a lightweight DL model is developed to detect the garbage successfully with a higher accuracy rate. The proposed lightweight DL model uses two activation functions named MISH and rectified linear unit to enhance the feature extraction and detect the garbage. Moreover, a multiaccess edge computing (MEC) server is allocated in the proposed scheme to improve the Quality of Service (QoS) (i.e., reduce latency and improve security). Furthermore, a blockchain-based secure hazardous garbage (e.g., infectious, toxic, or radioactive materials) tracking technique is concluded in this scheme to identify the individual and reduce the potential harm to the environment. Experimental results demonstrate that the UxV can successfully detect the garbage using the proposed lightweight DL model within a minimum time frame and the obtained accuracy is higher than the other existing DL models. Besides, QoS has been investigated to verify the efficacy of the proposed scheme. Finally, a private blockchain network is established to demonstrate the performance of the proposed hazardous garbage tracking technique.
Due to aggressive urbanization (with population size), waste increases exponentially, resulting in environmental damage. Even though it looks challenging, such an issue can be controlled if we can reuse them. To handle this, in our work, we design a machine learning and blockchain-oriented system that identifies thewaste objects/products and recommends to the user multiple ’Do-It-Yourself’ (DIY) ideas to reuse or recycle. Blockchain records every transaction in the shared ledger to enable transaction verifiability and supports better decision-making. In this study, a Deep Neural Network (DNN) trained on about 11700 images is developed using ResNet50 architecture for object recognition (training accuracy of 94%).We deploy several smart contracts in the Hyperledger Fabric (HF) blockchain platform to validate recommended DIY ideas by blockchain network members. HF is a decentralized ledger technology platform that executes the deployed smart contracts in a secured Docker container to initialize and manage the ledger state. The complete model is delivered on a web platform using Flask, where our recommendation system works on a web scraping script written using Python. Fetching DIY ideas using web-scraping takes nearly 1 second on a desktop machine with an Intel Core-i7 processor with 8 cores, 16 GB RAM, installed with Ubuntu 18.04 64-bit operating system, and Python 3.6 package. Further, we evaluate blockchain-based smart contracts’ latencies and throughput performances using the hyperledger caliper benchmark. To the best of our knowledge, this is the first work that integrates blockchain technology and deep learning for the DIY recommender system.
Deep learning techniques with neural networks are developing prominently in recent years and have been deployed in numerous applications. Despite their great success, in many scenarios it is important for the users to validate that the inferences are truly computed by legitimate neural networks with high accuracy, which is referred to as the integrity of machine learning predictions. To address this issue, in this paper, we propose zkCNN, a zero knowledge proof scheme for convolutional neural networks (CNN). The scheme allows the owner of the CNN model to prove to others that the prediction of a data sample is indeed calculated by the model, without leaking any information about the model itself. Our scheme can also be generalized to prove the accuracy of a secret CNN model on a public dataset.