ESEM 2026 Overview

This year we had a paper published at the 2026 International Symposium on Empirical Software Engineering and Measurement. Unfortunately we were unable to attend the conference this year, but this post summarizes our contribution:

Do Smart Contract Auditing Results Transfer Across Datasets? A Two-Benchmark Empirical Study of Static and LLM-Based Security Tools

Shawal Khalid, Joaquin Tuckett, Chris Brown

• 🔍 Problem: Smart contracts are a foundational software component of modern blockchain ecosystems, yet they are immutable after deployment and operate in economically incentivized environments. Prior work introduces tools across to detect security vulnerabilities in smart contracts, however we lack insights on how the performance of different auditors transfers across paradigms and datasets​.​

• 🧪 Study: We conducted benchmarking study comparing security-based smart contract analyzers across different paradigms (static/rule-based, symbolic execution, logic-based, and LLM-based) across two datasets (SmartBugs and FORGE).

• 📊 Findings: Tool performance did not transfer cleanly across datasets, with major drops in performance on the real-world smart contracts in FORGE compared to the controlled samples in SmartBugs. Approaches also varied on the same datasets across F1, recall, and precision results. We also observed category-level blind spots across vulnerability categories.

• 💡 Implication: We discuss implications for smart contract auditing in practice and research, motivating the need for hybrid analysis approaches, cross-benchmark analyses of tools, and category-dependent evaluations.

While we were not able to attend this year, we hope to continue contributing and attend this conference again in the future. If you are interested in this work and would like to learn more, please feel free to reach out