KOPA-Bench: Benchmarking Multi-Step Tool-Calling over Korean Open Public APIs with EDGE

Forum topic · 小凯 · 2026-09-09

Summary

This arXiv paper (2609.05395) by Dain Kim, Eungi Cho, Kyumin Kim, Shinyeong Noh, and Kyuseong Lim addresses a key gap in LLM tool-calling. Data-sovereignty regulations increasingly push public institutions to deploy open-source, on-premise LLM agents that chain multiple tool-calls across live government APIs, yet open-source models consistently underperform in this multi-step setting and no benchmark measured that gap. The authors introduce KOPA-Bench (Korean Open Public API Benchmark), comprising 145 real-world tasks over live Korean government APIs. They also propose EDGE (Execution-grounded Dynamic Graph), a data synthesis method driven by live execution: it builds a graph of how each tool's output can feed another's input, keeps only links that succeed when actually called against the live APIs, and traverses these verified links to synthesize executable multi-step trajectories. Fine-tuning a 9B model with GRPO on the synthesized dataset brought it close to its unfine-tuned 27B sibling on both KOPA-Bench and BFCL, showing execution-grounded synthesis can substantially narrow the capability gap for on-premise agents.

Paper Overview

Field: NLP
Authors: Dain Kim, Eungi Cho, Kyumin Kim, Shinyeong Noh, Kyuseong Lim
Published: 2026-09-04
arXiv: 2609.05395

Abstract (translated)

Data-sovereignty regulations increasingly require public institutions to deploy open-source, on-premise LLM agents that chain multiple tool-calls across live government APIs. However, open-source models consistently underperform in this multi-step setting, and no existing benchmark measures the gap.

The paper introduces the Korean Open Public API Benchmark (KOPA-Bench), comprising 145 real-world tasks built on live Korean public APIs.

To close the capability gap, the authors present EDGE (Execution-grounded Dynamic Graph), a data synthesis method for tool-calling driven by live execution:

1. Build a graph of how each tool's output can feed another tool's input. 2. Keep only the links that succeed when actually called against the live APIs. 3. Traverse these verified links to synthesize executable multi-step trajectories.

Results

A 9B model fine-tuned via GRPO on the EDGE-synthesized dataset substantially narrows the gap to its unfine-tuned 27B sibling, improving performance on both KOPA-Bench and the BFCL benchmark. This demonstrates that execution-grounded dynamic graph data synthesis is an effective way to make smaller open-source models viable for on-premise, multi-step government API agents.