KOPA-Bench: Benchmarking Multi-Step Tool-Calling over Korean Open Public APIs with the EDGE Data Synthesis Method

Forum topic · 小凯 · 2026-09-09

Summary

Data-sovereignty regulations increasingly push public institutions to deploy open-source, on-premise LLM agents that chain multiple tool calls across live government APIs, yet open-source models underperform in this multi-step setting and no benchmark measures the gap. This paper introduces KOPA-Bench (Korean Open Public API Benchmark), comprising 145 real-world tasks over live Korean public APIs. To close the gap, the authors propose EDGE (Execution-grounded Dynamic Graph for tool-calling data synthesis), which builds a graph of how each tool's output can feed another tool's input, keeps only links that succeed when actually invoked against live APIs, and traverses these verified links to synthesize executable multi-step trajectories. Fine-tuning a 9B model via GRPO on the synthesized dataset brings performance close to an unfine-tuned 27B model in the same family on both KOPA-Bench and BFCL. Authors: Dain Kim, Eungi Cho, Kyumin Kim, Shinyeong Noh, Kyuseong Lim. arXiv: 2609.05395.

Overview

Research area: NLP Authors: Dain Kim, Eungi Cho, Kyumin Kim, Shinyeong Noh, Kyuseong Lim arXiv: 2609.05395

Abstract (translated from the Chinese summary)

Data-sovereignty regulations increasingly require public institutions to deploy open-source, on-premise LLM agents that chain multiple tool-calls across live government APIs. However, open-source models consistently underperform in this multi-step setting, and no existing benchmark measures the gap.

This paper introduces the Korean Open Public API Benchmark (KOPA-Bench), comprising 145 real-world tasks over live Korean public APIs.

To close the gap, the authors present EDGE (Execution-grounded Dynamic Graph for tool-calling data synthesis), driven by live execution:

Builds a graph of how each tool's output can feed another tool's input
Keeps only the links that succeed when actually called against the live APIs
Traverses these verified links to synthesize executable multi-step trajectories

Fine-tuned via GRPO on the resulting dataset, their 9B model substantially approaches the performance of an unfine-tuned 27B model in the same family on both KOPA-Bench and the BFCL benchmark.

Original abstract (excerpt)

> Data-sovereignty regulations increasingly require public institutions to deploy open-source, on-premise LLM agents that chain multiple tool-calls across live government APIs. However, open-source models consistently underperform in this multi-step setting, and no existing benchmark measures the gap. We introduce the Korean Open Public API Benchmark (KOPA-Bench), comprising 145 real-world tasks. To close this gap, we present EDGE, an Execution-grounded Dynamic Graph for tool-calling data synthEsis driven by live execution...

---

*Collected automatically on 2026-09-09.*