A Survey on Employing Large Language Models for Text-to-SQL Tasks (ACM Computing Surveys, May 2025)

This post introduces a survey titled "A Survey on Employing Large Language Models for Text-to-SQL Tasks", published in *ACM Computing Surveys* in May 2025.

Publication Information

| Field | Content | |-------|---------| | Title | A Survey on Employing Large Language Models for Text-to-SQL Tasks | | Venue | ACM Computing Surveys, May 2025 | | Source | https://dl.acm.org/doi/abs/10.1145/3737873 | | Type | Survey | | Section | Querying Structured Information |

Overview

The survey systematically reviews the intersection of large language models (LLMs) and Text-to-SQL — the task of translating natural-language questions into SQL queries over structured databases. It addresses how to redraw the responsibility boundaries among retrieval, ranking, generation, and tool invocation in the LLM era, with reasoning budget and action space (whether to retrieve, how many retrieval steps, which tools to call) as the new design variables.

Key Points

• Provides a unified perspective that organizes scattered related work into a comparable framework.
• Decomposes method components (representation learning, retrievers, re-rankers, planners, generators, feedback mechanisms) for easier engineering adoption.
• Offers reproducible benchmarks, datasets, and classification tables that lower the entry cost for new researchers.
• Discusses interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration, and paths from research prototypes to industrial systems.
• Identifies open problems: evaluation trustworthiness, latency and cost, hallucination and safety, and cross-lingual / multimodal extension.

Taxonomy

| Dimension | Subclass | Representative Ideas | Strengths | Limitations | |-----------|----------|----------------------|-----------|-------------| | Modeling paradigm | Discriminative / generative retrieval | Bi-encoder, cross-encoder, DSI, GPT-indexed | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization objectives | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Annotation scarcity | | Evaluation | Offline / online / human | nDCG, MRR, LLM-as-judge, A/B testing | Comparable | Diverges from real satisfaction |

Research Timeline

• 2019–2021: BERT re-ranking and DPR establish the foundations of neural retrieval.
• 2022–2023: RAG and FreshLLMs drive retrieval–generation fusion.
• 2024 onward: Conversational/agentic search and generative recommender systems (Gen-RecSys) explode.
• 2025–2026: RL-trained search agents, Deep Research, and GraphRAG become new growth fronts.

Evaluation Paradigms

Typical benchmarks and metrics covered by surveys in this area:

• Datasets: MS MARCO, BEIR, Natural Questions, domain corpora, public recommendation sets
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, token cost
• Baselines: BM25, dense retrieval, cross-encoder re-ranking, retrieval-free LLMs, commercial search APIs

Specific quantitative results should be verified against the original PDF.

Insights for Search / Recommendation / Personalization

1. Architecture: Cascaded retrieve–rerank–generate remains mainstream, but the agentic paradigm turns retrieval count and policy itself into learnable objects. 2. Data: High-quality instruction data and click/session logs are both critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation. 4. Product: Latency, cost, explainability, and safety are hard constraints in industrial deployment — do not optimize academic benchmarks alone.

Open Problems and Future Directions

Commonly cited gaps include the lack of unified benchmarks, non-reproducible private data, LLM evaluation bias, and the safety/cost constraints of agentic systems. Future directions include finer-grained process supervision, joint retrieval–reasoning training, enterprise metadata governance, and multimodal / cross-lingual consistency.

Engineering Checklist

| Item | Question | Recommendation | |------|----------|----------------| | Data | PII in training/index? Versioning? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | p99 budget? Retrieval steps? | Cascades with early stopping, query caching, async re-ranking | | Quality | Does offline gain translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation checks | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelisting, adversarial detection, output filtering | | Cost | Per-query tokens and GPU usage? | Small-model routing, distillation, hybrid sparse+dense |

Related Cross-References

• A Survey of Text-to-SQL in the Era of LLMs (arXiv:2408.05109)
• Large Language Model for Table Processing: A Survey (Frontiers, Jan 2025)
• Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL (arXiv:2406.08426)
• Querying Databases with Function Calling (arXiv:2502.00032, Jan 2025)

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain | | Agentic Search | Modeling search as sequential decision-making with tool calls | | Gen-IR | Generative Information Retrieval |

> Note: Quantitative conclusions should be verified against the original publication at https://dl.acm.org/doi/abs/10.1145/3737873.