KOPA-Bench: Benchmarking Multi-Step Tool-Calling over Korean Open Public APIs with Execution-Grounded Data Synthesis

Forum topic · 小凯 · 2026-09-09

Summary

This paper introduces the Korean Open Public API Benchmark (KOPA-Bench), a benchmark of 145 real-world tasks for evaluating multi-step tool-calling by LLM agents over live government APIs. Motivated by data-sovereignty regulations that push public institutions toward open-source, on-premise LLM agents, the authors observe that open-source models consistently underperform when chaining multiple tool calls across live APIs, and that no existing benchmark measures this gap. To address it, they propose EDGE, an Execution-grounded Dynamic Graph method for synthesizing tool-calling data. EDGE constructs a graph of how each tool's output can feed another tool's input, retains only links verified by real API calls, and traverses these validated links to produce executable multi-step trajectories. Fine-tuning a 9B open-source model with GRPO on the synthesized dataset substantially narrows the gap to an untuned 27B model from the same family on both KOPA-Bench and the BFCL benchmark. Paper: arXiv 2609.05395, by Dain Kim, Eungi Cho, Kyumin Kim, Shinyeong Noh, and Kyuseong Lim.

Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and Execution-Grounded Data Synthesis (EDGE)

Field: NLP Authors: Dain Kim, Eungi Cho, Kyumin Kim, Shinyeong Noh, Kyuseong Lim Published: 2026-09-04 arXiv: 2609.05395

Abstract (translated)

As data-sovereignty regulations increasingly require public institutions to deploy open-source, on-premise LLM agents that chain multiple tool-calls across live government APIs, open-source models consistently underperform in this multi-step setting, and no existing benchmark measures the gap. This paper introduces the Korean Open Public API Benchmark (KOPA-Bench), comprising 145 real-world tasks. To close this gap, the authors present EDGE — an Execution-grounded Dynamic Graph for tool-calling data synthesis driven by live execution. EDGE builds a graph of how each tool's output can feed another's input, keeps only the links that succeed when actually called against the live APIs, and traverses these verified links to synthesize executable multi-step trajectories.

Key Results

KOPA-Bench: 145 real-world tasks built on live Korean government open public APIs, the first benchmark targeting multi-step tool-calling in this setting.
EDGE data synthesis: Constructs a tool-input/output dependency graph; only edges verified by real API execution are kept, and validated links are traversed to generate executable multi-step trajectories.
GRPO fine-tuning: A 9B model fine-tuned via GRPO on the EDGE-synthesized dataset substantially approaches the performance of an untuned 27B model from the same family on both KOPA-Bench and the BFCL benchmark.

Original Abstract

> Data-sovereignty regulations increasingly require public institutions to deploy open-source, on-premise LLM agents that chain multiple tool-calls across live government APIs. However, open-source models consistently underperform in this multi-step setting, and no existing benchmark measures the gap. We introduce the Korean Open Public API Benchmark (KOPA-Bench), comprising 145 real-world tasks. To close this gap, we present EDGE, an Execution-grounded Dynamic Graph for tool-calling data synthEsis driven by live execution. EDGE builds a graph of how each tool's output can feed another's input, keeps only the links that succeed when actually called against the live APIs, and traverses these verified links to synthesize executable multi-step trajectories. Fine-tuned via GRPO on the resulting dataset, our 9B model substantially approaches the untuned 27B model of the same family on both KOPA-Bench and BFCL.

*Auto-collected on 2026-09-09.*