Build2SPARQL: A Large-Scale Text-to-SPARQL Benchmark for Building Knowledge Graphs

Forum topic · 小凯 · 2026-10-05

Summary

Build2SPARQL (arXiv:2610.00224) is a large-scale benchmark for translating natural-language questions into SPARQL over building automation knowledge graphs encoded with ontologies such as Brick and ASHRAE 223P. The dataset is generated by a KG-grounded pipeline in which SPARQL queries are produced and validated entirely by graph-traversal code, while LLMs generate only the natural-language questions, keeping query correctness independent of model behavior. The pipeline mines six query pattern families—linear chains, branching, UNION, aggregation, OPTIONAL, and property filters—expressed in five lexical registers. Applied to 201 building knowledge graphs (180 Brick, 21 ASHRAE 223P), it yields 6,136 executable SPARQL queries and 30,680 questions. Human validation on 300 samples showed 98.8% semantic fidelity, 98.8% naturalness, and 84.0% operational plausibility. Retrieval-augmented evaluation of three open-weight models lifted exact-match scores from 0.2–20% (zero-shot) to 56–65% (three-shot with retrieval).

Paper Overview

Field: Machine Learning Author: Wooyoung Jung Published: 2026-10-05 arXiv: 2610.00224

Abstract

Building automation systems are increasingly represented as semantic knowledge graphs (KGs) using ontologies such as Brick and ASHRAE 223P, creating a machine-readable substrate for artificial-intelligence applications. One promising application is translating natural-language questions into SPARQL (text-to-SPARQL), which would let building operators query these graphs through language agents, but progress is limited by the scarcity of large natural-language/SPARQL benchmarks.

This paper presents Build2SPARQL, a large-scale benchmark for building KGs generated by a KG-grounded pipeline: SPARQL queries are produced and validated entirely by graph-traversal code, while large language models generate only the natural-language questions, keeping query correctness independent of model behavior.

Key Points

The pipeline mines six query pattern families: linear chains, branching, UNION, aggregation, OPTIONAL, and property filters, each expressed in five lexical registers.
Applied to 201 building knowledge graphs (180 Brick, 21 ASHRAE 223P), producing 6,136 executable SPARQL queries and 30,680 questions.
Human validation on 300 samples: 98.8% semantic fidelity, 98.8% naturalness, 84.0% operational plausibility (two annotators).
Retrieval-augmented evaluation of three open-weight models raised exact-match from 0.2–20% (zero-shot) to 56–65% (three-shot with retrieval).

---

*Auto-collected on 2026-10-05*

Source: arXiv:2610.00224