HarnessSQL: Harness-Native Training Fixes the Train-Deploy Mismatch in SQL Agents

Forum topic · ✨步子哥 · 2026-10-09

Summary

HarnessSQL addresses the train-deployment mismatch in Text-to-SQL agents: models trained on static 'question + schema to SQL' pairs fail when deployed in live, stateful database environments. The paper identifies three layers of mismatch—environmental (static schemas vs. live databases), harness (absence vs. presence of an interaction protocol), and supervision (final-answer vs. full-trajectory correctness). Its solution trains models inside the harness itself: (1) build executable sandboxes from 79 real databases with runtime schemas, sample values, and valid JOIN paths; (2) define a minimal four-tool SQL interaction protocol and collect 2,512 verified teacher trajectories filtered by denotational verification and protocol checks; (3) apply full-sequence SFT followed by execution-reward RL. On Spider 2.0-SQLite, Qwen3-8B improves from 15.5% to 45.2% and Qwen3-14B from 22.2% to 54.8%, far exceeding larger specialized models like OmniSQL-32B (13.3%). Trained models also transfer to unseen interactive benchmarks (BIRD-Interact Mini, LiveSQLBench), suggesting acquisition of general database-interaction ability. The post also discusses limitations: small data scale, SQLite-only experiments, and teacher-model dependence.

An Overlooked Rupture

You train a Text-to-SQL model on "question + schema → SQL query" pairs. It performs well on Spider benchmarks—15% accuracy on Spider 2.0-SQLite. Then you deploy it. A user asks which customers spent over 100K last quarter, the model generates SQL, but the real database's schema differs from training: abbreviated table names, ambiguous columns, non-shortest JOIN paths. The SQL errors out.

In the traditional paradigm, that's the end—the model only learned to "generate SQL," never "what to do when SQL fails." A real database agent should be able to run probe queries, diagnose errors, revise SQL, and resubmit. This is the train-deployment mismatch HarnessSQL tackles: training saw one-shot SQL generation, but deployment requires stateful multi-turn interaction with a database.

Three Layers of Mismatch

1. Environment mismatch. Training databases are static schema descriptions; deployed databases are live—with runtime state, real data distributions, and specific JOIN paths. 2. Harness mismatch. The "harness" is the interaction protocol between agent and database (what operations are allowed, what feedback is returned, when to submit a final answer). Training had no harness; deployment suddenly has one—like handing someone who never drove a car with pedals and saying "you'll figure it out." 3. Supervision mismatch. Training supervises "is the final SQL correct"; deployment requires learning whether the entire interaction trajectory is correct—when to probe, when to revise, when to submit.

The HarnessSQL Approach

Core idea: let the model play inside the harness during training.

Step 1: Executable sandboxes. For each target database, extract runtime schema, sample column values, and valid JOIN paths from a real engine. 79 databases, each with a hidden execution oracle.
Step 2: Four-tool SQL harness. A minimal protocol where the agent interacts with the database through only four tools. A teacher model runs trajectories; only those passing denotational verification (correct execution results) and protocol checks are kept—yielding 2,512 verified trajectories.
Step 3: Full-trajectory SFT + execution-reward RL. Full-sequence SFT on verified trajectories (learning the whole interaction, not just final SQL), followed by RL with execution results as reward.

The Numbers

| Model | Before | After HarnessSQL | Gain | |-------|--------|------------------|------| | Qwen3-8B | 15.5% | 45.2% | +29.7pp | | Qwen3-14B | 22.2% | 54.8% | +32.6pp |

On Spider 2.0-SQLite, that's nearly a 3x improvement. More importantly, cross-benchmark transfer: models trained with HarnessSQL also perform well on BIRD-Interact Mini and LiveSQLBench—benchmarks never seen in training—showing they learned general "interacting with databases" ability, not memorization.

For comparison, specialized SQL models like OmniSQL-32B (13.3%) and Arctic-Text2SQL-32B (14.8%) score only 13–15% on Spider 2.0-SQLite—4x the parameters, one-third the accuracy.

Why It Matters

1. Train-deploy mismatch is universal for agents. Code models trained on "description → code" must learn "read error → fix → rerun"; tool-use models trained on "task → call" must learn "call fails → retry." Every agent has a static-training / dynamic-deployment rupture. 2. Harness-native training is more fundamental than agent frameworks. Most compensation happens at inference time—ReAct prompts, reflection, retry logic. HarnessSQL's insight: rather than compensating at inference, train inside the harness from the start. A shift from "framework-level patches" to "training-level capability." 3. Minimalism of the four-tool protocol. Like Unix philosophy, a small, precise interface beats a sprawling one. The four-tool protocol is the "narrow waist" of SQL interaction.

An Analogy

Traditional Text-to-SQL training is like teaching someone to write cover letters by showing examples—but a real job search involves researching the company, checking LinkedIn, writing, waiting, revising. HarnessSQL trains the whole process, not just the letter-writing step.

Honest Assessment

Data scale. 79 databases and 2,512 trajectories is small for a general database agent; real scenarios involve multi-table JOINs, window functions, CTEs, stored procedures. Coverage needs more validation.
SQLite limitation. Main experiments are SQLite-only; transfer to PostgreSQL/MySQL-style server databases isn't fully verified.
Teacher dependence. Trajectories come from a teacher model, whose capability ceiling caps the student's.

Still, as a proof of concept for train-deployment mismatch, HarnessSQL is persuasive. It points toward an emerging paradigm: agent training is no longer input→output mapping, but trajectory learning through environment interaction.

---

Paper: HarnessSQL: Harness-Native Training for SQL Agents in Realistic Database Environments

Code: YangHaolin0526/HarnessSQL

Related benchmarks:

bird-bench/livesqlbench (LiveSQLBench)
Spider 2.0