A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers (Oct 2025, arXiv)
Overview
- Title: A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers
- Authors / Affiliations: Ming Hu, Chenglong Ma, Wei Li, Wanghan Xu, Jiamin Wu, Jucheng Hu, et al. (~120 authors in total)
- Link: https://arxiv.org/abs/2508.21148
- Type: Survey
- Category: Deep Research
- A unified perspective that brings scattered related work into a comparable framework.
- A clear decomposition of method components: representation learning, retrievers, rerankers, planners, generators, and feedback mechanisms.
- Reproducible benchmarks, datasets, and taxonomy tables that lower the entry barrier for follow-up research.
- Discussion of interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration, including migration paths from research prototypes to industrial systems.
- Explicit open problems: evaluation trustworthiness, latency and cost, hallucination and safety, and cross-lingual / multimodal scaling.
- Dense retrieval: high recall and low latency, suited to first-stage retrieval.
- Late interaction (e.g., ColBERT): higher precision but larger indexes.
- Generative IR: directly "generating" documents via tokens or docids, simplifying cascades.
- Agentic search: modeling search as sequential decision-making, supporting multi-hop reasoning and self-reflection.
- 2019–2021: BERT reranking and DPR establish neural retrieval foundations.
- 2022–2023: RAG and FreshLLMs drive retrieval–generation fusion.
- 2024 onward: conversational/agentic search and Gen-RecSys proliferate.
- 2025–2026: RL-trained search agents, Deep Research, and GraphRAG become new growth frontiers.
- Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora, public recommendation sets.
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency and token cost.
- Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs.
- Ablations: contribution of retrieval steps, reranking depth, and training-data scale.
- IR: Information Retrieval
- RAG: Retrieval-Augmented Generation
- LTR: Learning to Rank
- nDCG: Normalized Discounted Cumulative Gain
- Agentic Search: framing search as sequential decision-making and tool invocation
- Gen-IR: Generative Information Retrieval
- A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications
- A Survey of LLM-based Deep Search Agents
- Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents
- AgentIR: Reasoning-Aware Retrieval for Deep Research Agents
- Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning
- BioMedArena: An Open-source Toolkit for Building and Evaluating Biomedical AI
- Original paper: A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers. arXiv:2508.21148
Background and Motivation
In large-scale search, recommendation, and personalization systems, agentic search has long faced challenges in efficiency, scalability, and user-intent understanding. Traditional pipeline approaches often treat retrieval, ranking, and generation as disjoint stages, making it hard to meet LLM-era demands for natural-language interaction, multi-hop reasoning, and real-time knowledge. This survey aims to systematically map the theoretical and practical boundaries of this intersection, covering open-domain information access, enterprise knowledge retrieval, conversational search, semantic understanding in recommender systems, and end-to-end architectures that coordinate external knowledge sources with generative models.
Key Contributions
Taxonomy
| Dimension | Subclasses | Representative Ideas | Strengths | Limitations | |---|---|---|---|---| | Modeling paradigm | Discriminative / generative retrieval | Bi-encoders, cross-encoders, DSI, GPT-indexed corpora | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization objective | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Scarce labels | | Evaluation | Offline / Online / Human | nDCG, MRR, LLM-as-judge, A/B tests | Comparable | Divergence from real satisfaction |
Main Research Lines
The field is typically organized into four threads:
Timeline of Evolution
Evaluation Paradigms
Typical setups covered by the survey include:
Quantitative results should be verified against the original PDF tables.
Key Takeaways for Search / Rec / Personalization
1. Architecture: cascade retrieval + rerank + generation remains mainstream, but agentic paradigms make "how many retrievals and what policy" itself learnable. 2. Data: high-quality instruction data and click/session logs are both critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation against human evaluation. 4. Productization: latency, cost, explainability, and safety policies are hard constraints in industry—not just academic benchmark scores.
Open Problems and Future Directions
Authors commonly cite: insufficient unified benchmarks, non-reproducible private data, LLM evaluation bias, and safety/cost constraints on agentic systems. Future work includes finer-grained process supervision, joint retrieval–reasoning training, enterprise metadata governance, and multimodal / cross-lingual consistency.
Limitations
Possible limitations include experiment scale constrained by GPU budgets, mismatch between benchmarks and real user distributions, English-centric data leaving cross-lingual generalization unknown, and safety risks of agentic systems on the open web. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs / structured databases, and causal and fairness constraints for recommender systems.
Engineering Checklist
| Item | Question | Suggestion | |---|---|---| | Data | PII in training/index? Version management? | Partitioned indexes, anonymization, rollback-able embedding versions | | Latency | p99 budget? Retrieval steps? | Cascade + early stopping, hot-query caching, async reranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation checks | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelisting, adversarial detection, output filtering | | Cost | Per-query tokens and GPU usage? | Route to smaller models, distillation, hybrid sparse+dense |