A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers (Oct 2025, arXiv)
Meta Information
| Field | Content | |-------|---------| | Title | A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers | | Authors / Affiliations | Ming Hu, Chenglong Ma, Wei Li, Wanghan Xu, Jiamin Wu, Jucheng Hu, et al. (120 authors in total) | | Published | October 2025 (arXiv) | | Link | https://arxiv.org/abs/2508.21148 | | Type | Survey | | Section | Deep Research |
One-line Summary
A comprehensive survey of scientific large language models (Sci-LLMs), tracing the field from data foundations to agentic frontiers such as tool use and autonomous research agents.
Background and Motivation
The survey addresses the rapidly growing intersection of LLMs and scientific research. As LLMs move from general-purpose chat systems toward domain-specific applications in science, questions around training data quality, domain adaptation, evaluation, and agentic capabilities (tool calling, multi-step reasoning, autonomous experimentation) become central. This work aims to systematically organize the theory and practice of this cross-disciplinary field.
Key Contributions
- Provides a unified taxonomy that brings scattered work on scientific LLMs into a comparable framework.
- Decomposes method components (data curation, representation learning, retrievers, planners, generators, feedback mechanisms) for easier engineering adoption.
- Offers reproducible benchmarks, datasets, and classification tables, lowering the entry barrier for new researchers.
- Discusses interfaces with emerging paradigms: LLM tool use, reinforcement learning, multi-agent collaboration, and paths from research prototypes to production systems.
- Lists open problems: evaluation trustworthiness, latency and cost, hallucination and safety, cross-lingual and multimodal scaling.
- 2019–2021: BERT re-ranking and DPR lay the foundation for neural retrieval.
- 2022–2023: RAG and FreshLLMs drive retrieval-generation fusion.
- 2024 onward: Conversational/agentic search and generative recommender systems (Gen-RecSys) explode.
- 2025–2026: RL-trained search agents, Deep Research systems, and GraphRAG become new growth frontiers.
- Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora.
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, token cost.
- Baselines: BM25, dense retrieval, cross-encoder re-ranking, retrieval-free LLMs, commercial search APIs.
- Ablations: contribution of retrieval steps, re-ranking depth, and training data scale.
- A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications
- A Survey of LLM-based Deep Search Agents
- Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents
- Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning
- Original paper: A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers, arXiv 2508.21148.
Taxonomy
| Dimension | Subclasses | Representative Approaches | Strengths | Limitations | |-----------|-----------|---------------------------|-----------|-------------| | Modeling paradigm | Discriminative retrieval / Generative retrieval | Bi-encoders, cross-encoders, DSI, GPT indexing | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calling | Flexible, interpretable | Latency, error propagation | | Optimization objectives | Relevance / Diversity / Freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Scarce annotations | | Evaluation | Offline / Online / Human | nDCG, MRR, LLM-as-judge, A/B testing | Comparable | Diverges from real satisfaction |
Research Timeline and Evolution
Evaluation Paradigms
Typical benchmarks and trends covered:
For exact quantitative results, refer to the tables in the original PDF.
Key Takeaways
1. Architecture: Cascaded retrieval + re-ranking + generation remains mainstream, but the agentic paradigm makes "when and how often to retrieve" a learnable decision. 2. Data: High-quality instruction data and click/session logs are equally critical; synthetic data must guard against leakage and distribution shift. 3. Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation. 4. Productization: Latency, cost, interpretability, and safety are hard constraints for industrial deployment—not just academic benchmarks.
Open Problems and Future Directions
The authors highlight: insufficient unified benchmarks, non-reproducible private data, LLM evaluation bias, and safety/cost constraints for agentic systems. Future directions include finer-grained process supervision, joint retrieval-reasoning training, enterprise metadata governance, and multimodal/cross-lingual consistency.
Limitations
Possible limitations include experiment scale constrained by GPU budgets, mismatch between benchmarks and real user distributions, English-centric data limiting cross-lingual generalization, and safety risks of open-web agent systems.