QuantumSemEval

Benchmarking LLMs' Understanding of Quantum Programs via Semantic Equivalence Checking

Deep Learning for Code (DL4C) Workshop @ International Conference on Machine Learning (ICML) 2026

Niloy Kumar Mondal

Bangladesh University of Engineering and Technology (BUET)

Corresponds to: nkm2105044@gmail.com

Abstract

Large language models (LLMs) have shown promise in reasoning about code, yet their ability to perform semantic reasoning over quantum programs remains largely unexplored. To address this gap, we introduce QuantumSemEval, a benchmark designed to evaluate LLMs' capacity to determine semantic equivalence between pairs of quantum programs, comprising 260 labeled pairs (130 equivalent and 130 non-equivalent) across diverse algorithms. We evaluate fifteen LLMs spanning six proprietary and nine open-weight architectures. While models reliably detect non-equivalence, identifying semantically equivalent circuits is substantially more challenging. Top performance on equivalent pairs reaches only 70.8%, with the majority of models performing below the 50% random baseline. These results highlight the current limitations of LLMs in semantic reasoning and provide a foundation for future work aimed at enhancing their understanding of quantum programs. The dataset and code are publicly available at quitttcat.github.io/QuantumSemEval2026.

Introduction

Can large language models truly understand the behavior of a quantum program? Quantum programs behave very differently from classical ones: their outputs are inherently probabilistic, even small modifications may leave behavior unchanged or produce substantially different results, and noise in current NISQ-era hardware further obscures whether an observed difference reflects a real change in behavior or just hardware noise — a task that is challenging even for human experts. Most existing quantum-programming benchmarks focus on whether an individual program runs and produces the expected output; they do not test whether two programs that may look different actually compute the same function. To address this gap, we introduce QuantumSemEval, a benchmark specifically designed to evaluate whether LLMs can determine semantic equivalence between pairs of quantum programs — covering 260 circuit pairs across a diverse set of quantum algorithms, transformations, and ZX-calculus optimizations. Our evaluation of fifteen LLMs reveals a pronounced asymmetry: models reliably detect non-equivalence, but reliably confirming equivalence remains a genuinely hard, largely unsolved problem — the top model reaches just 70.8%, with most models performing at or below the 50% random baseline.

Dataset Overview

Equivalent pairs are generated by applying ZX-calculus simplification (PyZX full_reduce) to a source circuit and pairing the original with its reduced form, verified via statevector fidelity F(ψ,ϕ)=|⟨ψ|ϕ⟩|²=1. Non-equivalent pairs apply seven targeted semantic-breaking mutations to circuits drawn from the same pool, verified by confirming F<1 on at least one computational-basis input.

Leaderboard

Phase I (Equivalent) accuracy is recall — the fraction of equivalent pairs correctly identified as equivalent. Phase II (Non-Equivalent) accuracy is specificity — the fraction of non-equivalent pairs correctly rejected. Random baseline is 50% for both. Click a column header to sort.

Cite Us

@inproceedings{
mondal2026quantumsemeval,
title={QuantumSemEval: Benchmarking {LLM}s' Understanding of Quantum Programs via Semantic Equivalence Checking},
author={Niloy Kumar Mondal},
booktitle={Deep Learning for Code: Towards Human-Centered Coding Agents},
year={2026},
url={https://openreview.net/forum?id=zVeYUddqiz}
}

MIT License · QuitttCat/QuantumSemEval2026