Identification and characterization of inference bottlenecks and class imbalance in Transformer-based Smart Contract Auditors

Abstract

This research investigates critical challenges in Transformer-based smart contract auditing systems, with a specific focus on inference instability and class imbalance in CodeBERT-based binary vulnerability classification. Layer-2 blockchain networks introduce highly complex architectures that increase the risk of smart contract exploits, where traditional static analysis tools such as Slither often produce large volumes of noisy, rule-based alerts. Recent advancements in pre-trained Transformer models, particularly CodeBERT, have demonstrated strong capabilities in semantic code understanding and vulnerability detection. However, during deployment of a fine-tuned CodeBERT-base model, we observe significant performance and stability issues. Initial inference experiments show a 100% false-positive rate, primarily attributed to severe class imbalance in the slither-audited-smart-contracts dataset and model sensitivity to specific smart contract patterns such as raw Ether transfer functions. In addition, system-level execution profiling reveals a silent segmentation fault during model initialization. Further investigation using Windows OS logs identifies dependency conflicts between PyTorch and PyArrow (via Hugging Face Datasets), particularly related to C++ DLL load-order issues. Experimental analysis demonstrates that modifying dependency import order, prioritizing PyArrow initialization, and enforcing strict model.eval() state management significantly improves inference stability. These findings highlight important architectural and deployment considerations for Transformer-based blockchain security systems and provide practical insights for improving the robustness of automated smart contract auditing pipelines in Layer-2 Web3 ecosystems.