The Bitter Lesson of Tool Calling: Programmatic Tool Calling Outperforms JSON Across 14 Models

Forum topic · ✨步子哥 · 2026-08-07

Summary

A 2026 paper from PricewaterhouseCoopers (arXiv:2608.06370) systematically compares JSON-based tool calling with Programmatic Tool Calling (PTC), where the model writes Python that invokes tools as functions. On the Berkeley Function Calling Leaderboard v4 (BFCL v4), PTC matched or beat JSON in 11 of 14 models overall and 13 of 14 in parallel fan-out, with GPT-5.6 gaining 10.6 percentage points. PTC also resisted context rot where JSON lost 2.3% on average. The authors argue the advantage grows with model generation, echoing Sutton's bitter lesson: as models get stronger, more general methods (writing code) outpace hand-engineered schemas. Limitations: weak code models regress under PTC; real-world error handling is untested; debugging code is harder than debugging JSON.

Overview

A 2026 paper by Ishan Patel, Sahil Sen, Elias Lumer, and Vamse Kumar Subbiah (PricewaterhouseCoopers) systematically compares two paradigms for LLM tool use:

JSON tool calling (current standard): model emits one JSON object per call, awaits result, emits the next.
Programmatic Tool Calling (PTC): model writes a Python script that invokes tools as functions, enabling chaining, branching, and parallel fan-out in a single turn.

Before this study, no standard-benchmark comparison across multiple model generations existed.

Benchmark Setup

Benchmark: Berkeley Function Calling Leaderboard v4 (BFCL v4) — https://gorilla.cs.berkeley.edu/blogs/13_bfcl_v4.html
Models: 14 LLMs
Dimensions: single-call accuracy, chaining (A→B→C), parallel fan-out (ten concurrent calls), and context rot (irrelevant noise injected into long dialogues)

Key Results

11/14 models: PTC matched or exceeded JSON on the main BFCL v4 test.
10.6 pp largest single-model gain, observed in the GPT-5.6 family.
13/14 models: PTC matched or exceeded JSON in the parallel fan-out scenario.
2.3% average drop for JSON under context rot; PTC showed negligible degradation.

Why PTC Wins Structurally

Parallel fan-out: JSON requires the model to internally plan and emit many tool calls simultaneously; models frequently drop, duplicate, or mis-order them. PTC expresses the same intent as a list comprehension (results = [tool(x) for x in inputs]), letting the Python interpreter handle execution. The cognitive burden of "understanding parallelism" is offloaded to the language.
Chaining: JSON forces the model to track state across turns manually. PTC uses variable bindings (a = step1(); b = step2(a)), so state is managed by the language, not the model's working memory.
Context rot: In PTC, tool parameters live inside syntactic code structures and are insulated from conversational noise; JSON generation is more easily contaminated by prior turns.

The Bitter Lesson Framing

The title references Sutton's 2019 *The Bitter Lesson*: hand-engineered priors (features, schemas, domain rules) are eventually outpaced by general methods scaled with compute. The authors find that PTC's relative advantage *grows* with model generation (GPT-5.6 > GPT-4.5 > GPT-4o), consistent with Sutton's prediction that stronger models benefit more from general-purpose approaches.

Honest Limitations

1. PTC helps only models with strong code abilities; three of fourteen models performed worse under PTC. 2. BFCL v4 is synthetic; real-world concerns (retries, timeouts, partial failures) are not measured. 3. Debugging Python scripts is harder than debugging JSON payloads. 4. The authors note no public code release accompanies the paper.

Position in the Lineage

2019 — Sutton, *The Bitter Lesson*: generality beats priors.
2024 — Anthropic productizes PTC.
2026 — This paper provides empirical confirmation that PTC dominates JSON on standard benchmarks, with the gap widening as models improve.