A Survey on Employing Large Language Models for Text-to-SQL Tasks (ACM Computing Surveys, May 2025)

Metadata

| Field | Value | |-------|-------| | Title | A Survey on Employing Large Language Models for Text-to-SQL Tasks, May 2025, ACM Computing Surveys | | Publication | ACM Computing Surveys, May 2025 | | Source | https://dl.acm.org/doi/abs/10.1145/3737873 | | Resource type | Survey | | Category | Querying Structured Information |

One-sentence Summary

This work systematically surveys the use of Large Language Models (LLMs) for Text-to-SQL tasks, consolidating a fast-growing body of research into a comparable framework.

Background and Motivation

In large-scale search, recommendation, and personalization systems, surveys have long faced challenges around efficiency, scalability, and user intent understanding. Traditional pipeline-style approaches often treat retrieval, ranking, and generation as disconnected stages, making it difficult to satisfy the combined demands of the LLM era: natural language interaction, multi-hop reasoning, and real-time knowledge. This survey, published in ACM Computing Surveys (May 2025), was proposed in this context, aiming to systematically map the theoretical and practical boundaries of this intersection.

The core scenarios include open-domain information access, enterprise knowledge retrieval, conversational search, semantic understanding in recommendation systems, and end-to-end architectures that coordinate external knowledge sources with generative models.

Core Contributions

• Provides or consolidates a unified perspective, bringing scattered related work into a comparable framework.
• Clearly decomposes method components (representation learning, retrievers, rerankers, planners, generators, feedback mechanisms) for easier engineering adoption.
• Offers reproducible benchmarks, datasets, and classification tables to lower the entry cost for subsequent researchers.
• Discusses interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration, outlining the path from research prototypes to industrial systems.
• Explicitly lists open problems: evaluation trustworthiness, latency and cost, hallucination and safety, cross-lingual and multimodal extension.

Taxonomy

| Dimension | Sub-category | Representative ideas | Strengths | Limitations | |-----------|--------------|----------------------|-----------|-------------| | Modeling paradigm | Discriminative / generative retrieval | Dual-tower, cross-encoders, DSI, GPT-indexing | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization objectives | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Scarce annotations | | Evaluation | Offline / Online / Human | nDCG, MRR, LLM-as-judge, A/B tests | Comparable | Deviates from real satisfaction |

Related Work Landscape

The survey typically places four main lines side by side: dense retrieval, late interaction, generative IR, and agentic search:

• Dense: high recall, low latency, suited to first-stage retrieval;
• Late interaction (e.g., ColBERT): higher accuracy but larger indexes;
• Generative IR: directly "generates" documents as tokens or docids, simplifying cascades;
• Agentic: models search as sequential decision-making, supporting multi-hop and self-reflection.

Timeline and Evolution

On the timeline, 2019–2021 saw BERT reranking and DPR lay the foundations of neural retrieval; 2022–2023 saw RAG and FreshLLMs drive retrieval-generation fusion; from 2024, conversational/agentic search and Gen-RecSys exploded; and 2025–2026 point to RL-trained search agents, Deep Research, and GraphRAG as new growth fronts.

Evaluation Paradigms

Typical benchmarks and trends covered include:

• Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora, public recommendation datasets;
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, and token cost;
• Baselines: BM25, dense retrieval, cross-encoder reranking, LLMs without retrieval, commercial search APIs;
• Ablations: verifying each module's (retrieval steps, reranking depth, training data scale) contribution to final quality.

For specific numerical results, consult the original PDF; this post generalizes the experimental design logic from abstracts and public metadata.

Key Insights

For the Search / Rec / Personalization community: 1. Architecture: cascade retrieve-rerank-generate remains mainstream, but agentic paradigms are turning "when and how often to retrieve" itself into a learnable object; 2. Data: high-quality instruction data and click/session logs are equally critical; synthetic data requires guarding against knowledge leakage and distribution shift; 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation; 4. Product: latency, cost, interpretability, and safety policies are hard constraints in industrial deployment—optimizing academic benchmarks alone is insufficient.

Open Problems and Future Directions

Commonly noted gaps include: insufficient unified benchmarks, unreproducible private data, LLM evaluation bias, and the safety/cost constraints of agentic systems. Future directions include finer-grained process supervision, joint retrieval-reasoning training, enterprise metadata governance, and multimodal and cross-lingual consistency.

Limitations

Possible limitations include: experiment scale constrained by GPU budgets, mismatch between benchmarks and real user distributions, unknown cross-lingual generalization due to English-centric data, and the safety risks of agentic systems on the open web. Future work may explore more efficient test-time compute allocation, deeper integration with knowledge graphs/structured databases, and causal/fairness constraints for recommendation systems.

Cross-references

• A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where ar…
• Large language model for table processing: a survey, Jan 2025, Frontie…
• Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL…
• Querying Databases with Function Calling, Jan 2025, arxiv

References

• Original: A Survey on Employing Large Language Models for Text-to-SQL Tasks, May 2025, ACM Computing Surveys. See the linked source.

Engineering Checklist

| Item | Question | Suggestion | |------|----------|------------| | Data | Does training/indexing contain PII? How are versions managed? | Partitioned indexes, anonymization, rollback-safe embedding versions | | Latency | What is the p99 budget? How many retrieval steps? | Cascade with early stopping, cache hot queries, async reranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelists, adversarial detection, output filtering | | Cost | Token and GPU usage per query? | Route to smaller models, distillation, sparse+dense hybrid |

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decision-making and tool invocation | | Gen-IR | Generative Information Retrieval |

Recommendations for Readers

1. Researchers: reproduce core comparisons; check whether statistical significance and compute costs are reported; 2. Engineers: extract pluggable modules (encoders, rerankers, planners) and evaluate integration cost with existing stacks; 3. Product managers: identify user-perceivable benefits (latency, answer trustworthiness, multi-turn consistency) rather than offline nDCG alone.