A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers (arXiv, Oct 2025)
Metadata
| Field | Content | |------|------| | Title | A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers | | Authors | Ming Hu, Chenglong Ma, Wei Li, Wanghan Xu, Jiamin Wu, Jucheng Hu, et al. (~120 authors) | | Link | https://arxiv.org/abs/2508.21148 | | Type | Survey | | Section | Deep Research |
Overview
This arXiv survey systematically reviews scientific large language models, from their data foundations to the agentic frontier where LLMs act as autonomous research and search agents. The forum post contextualizes the paper within the intersection of agentic search and large-scale search/recommendation systems.
Background and Motivation
Agentic search in large-scale search, recommendation, and personalization systems has long faced challenges in efficiency, scalability, and user intent understanding. Traditional pipeline approaches separate retrieval, ranking, and generation, which struggles to meet LLM-era demands for natural language interaction, multi-hop reasoning, and real-time knowledge. The survey aims to map the theory and practice of this intersection.
Core Contributions
- A unified perspective that organizes scattered related work into a comparable framework.
- Clear decomposition of method components (representation learning, retrievers, rerankers, planners, generators, feedback mechanisms).
- Reproducible benchmarks, datasets, and taxonomy tables that lower the entry barrier for new researchers.
- Discussion of interfaces with LLM tool calling, reinforcement learning, and multi-agent collaboration, including paths from research prototypes to industrial systems.
- Explicit open problems: evaluation trustworthiness, latency and cost, hallucination and safety, cross-lingual and multimodal scaling.
- Dense retrieval: high recall, low latency; suited to first-stage retrieval.
- Late interaction (e.g., ColBERT): higher accuracy but larger indexes.
- Generative IR: directly "generates" documents via tokens or docids, simplifying cascades.
- Agentic search: models search as sequential decision-making, supporting multi-hop reasoning and self-reflection.
- 2019–2021: BERT reranking and DPR establish neural retrieval foundations.
- 2022–2023: RAG and FreshLLM drive retrieval-generation fusion.
- 2024 onwards: conversational/agentic search and Gen-RecSys explode.
- 2025–2026: RL-trained search agents, Deep Research, and GraphRAG become new growth frontiers.
- Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora, public recommendation sets.
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, token cost.
- Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs.
- A Comprehensive Survey of Deep Research
- A Survey of LLM-based Deep Search Agents
- Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents
- AgentIR: Reasoning-Aware Retrieval for Deep Research Agents
- Agentic Reasoning
- BioMedArena
- Original paper: https://arxiv.org/abs/2508.21148
Taxonomy
| Dimension | Subclass | Representative ideas | Strengths | Limitations | |------|------|------|------|------| | Modeling paradigm | Discriminative / generative retrieval | Bi-encoders, cross-encoders, DSI, GPT indexing | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization goals | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Scarce labels | | Evaluation | Offline / Online / Human | nDCG, MRR, LLM-as-judge, A/B tests | Comparable | Gap with real satisfaction |
Four Main Research Lines
Timeline of Research Evolution
Evaluation Paradigms
Key Insights for Search / Rec / Personalization
1. Architecture: cascade retrieval + rerank + generation remains mainstream, but the agentic paradigm makes retrieval count and strategy themselves learnable. 2. Data: high-quality instruction data and click/session logs are both critical; synthetic data must guard against leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation. 4. Product: latency, cost, explainability, and safety are hard constraints for industrial deployment—do not optimize academic benchmarks alone.
Open Problems and Future Directions
Insufficient unified benchmarks, non-reproducible private data, LLM evaluation bias, and safety/cost constraints of agentic systems. Future work includes finer-grained process supervision, joint retrieval-reasoning training, enterprise metadata governance, and multimodal/cross-lingual consistency.
Engineering Checklist
| Item | Question | Suggestion | |--------|------|------| | Data | PII in training/index? Versioning? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | p99 budget? How many retrieval steps? | Cascade + early stopping, query caching, async reranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Security | Poisoning/bias from open retrieval? | Source whitelisting, adversarial detection, output filtering | | Cost | Token and GPU usage per query? | Small-model routing, distillation, hybrid sparse+dense |
Glossary
| Term | Meaning | |------|------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain | | Agentic Search | Modeling search as sequential decision-making and tool calling | | Gen-IR | Generative Information Retrieval |