TAHOE: Text-to-SQL with Automated Hint Optimization from Experience

Forum topic · 小凯 · 2026-06-12

Summary

TAHOE is a system that treats prompt optimization as a dynamic data management problem for production Text-to-SQL. It uses an error-driven hint learning pipeline that consolidates debugging traces into a structured Hint Bank: compiler feedback is distilled into reusable Syntax Hints for SQL dialect-specific rules, while execution and user feedback become Semantic Hints for schema- and user-specific logic. A policy layer models conflicting user intents as competing policies under shared natural-language triggers, with recency signals and post-learning attribution statistics. At inference, TAHOE retrieves relevant hints to guide an LLM through logical planning before SQL synthesis, without updating model parameters. On Spider 2.0-Snow with GPT-5.5 (113 supervised Spider 2.0-Snow-0212 examples), TAHOE raised pass rate from 61.95% to 79.42%, pass-at-4 from 72.57% to 87.61%, achieved 100% Snowflake syntax pass rate, and reduced average compiler-feedback critique rounds per candidate from 2.79 to 0.12. The same hint bank transfers to weaker backbones, including a 19.7-point pass-rate gain on Doubao-2.0-lite.

Paper Overview

Field: Machine Learning
Authors: Zhiyi Chen, Jie Song, Peng Li
Published: 2026-06-10
arXiv: 2606.12387

Abstract

Large Language Models (LLMs) have democratized database access through Text-to-SQL, but moving from prototypes to production remains difficult. Real deployments must handle strict SQL dialects, massive schemas, and evolving user preferences, while supervised fine-tuning is costly and rigid and agentic test-time scaling is expensive.

We present Tahoe, a system that treats prompt optimization as a dynamic data management problem. Tahoe uses an error-driven hint learning pipeline across Development and Deployment to consolidate debugging traces into a structured Hint Bank. Compiler feedback is distilled into reusable Syntax Hints for dialect-specific rules, while execution and user feedback are converted into Semantic Hints for schema- and user-specific logic.

Tahoe further introduces a Policy layer that models conflicting user intents as competing policies under shared natural-language triggers, with recency signals and post-learning attribution statistics summarizing empirical success, harm, inertia, and support.

At inference time, Tahoe retrieves relevant hints and guides the LLM through logical planning before SQL synthesis. The authors implement and evaluate the development-phase workflow, leaving deployment-time human-feedback updates to future work.

Key Results

On Spider 2.0-Snow, Tahoe substantially improves Text-to-SQL without updating model parameters:

Using GPT-5.5 on 113 supervised Spider 2.0-Snow-0212 examples:
Pass rate improved from 61.95% to 79.42%
pass-at-4 improved from 72.57% to 87.61%
Achieved 100% Snowflake syntax pass rate
Average compiler-feedback critique rounds per sampled candidate reduced from 2.79 to 0.12
The same hint bank transfers to weaker backbones, including a 19.7 percentage-point pass-rate improvement on Doubao-2.0-lite