BLADE: A Self-Instruct Synthetic Benchmark for Evaluating LLMs in the Blockchain Domain

Abstract

Although Large Language Models (LLM) have shown impressive performance across various domains, there is a shortage of benchmarks for systematically evaluating their in-depth understanding of specialized fields such as blockchain. This study extends the Self-Instruct methodology to introduce BLADE (Blockchain Large Language model Assessment Dataset for Evaluation), a comprehensive benchmark dataset for assessing LLM comprehension in the blockchain domain. BLADE consists of a total of 1,382 questions organized according to a systematic classification of blockchain knowledge, featuring a detailed structure with 15 main categories and 5 sub-categories for each. The benchmark covers the entire spectrum of blockchain knowledge, from its fundamentals to consensus mechanisms, architecture, smart contracts, token economy, Decentralized Finance (DeFi), NFT(Non-Fungible Token)s and digital assets, security, governance, and real-world application cases. In this research, we present a benchmark generation methodology utilizing the domain knowledge of GPT-4.5, which allowed us to create high-quality evaluation items of varying difficulty and types from expert-verified seed questions. The evaluation results of various open-source LLMs, including Qwen, DeepSeek, and Kanana, on BLADE showed that current models exhibit significant differences in their understanding of blockchain, with Qwen2.5-7B-Instruct-1M achieving the highest performance. The BLADE benchmark provides a tool for precisely evaluating and improving the blockchain comprehension of LLMs, thereby promoting the effective fusion of AI and blockchain technology and contributing to the development of more reliable decentralized systems.