A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers

This post summarizes a survey paper available on arXiv: https://arxiv.org/abs/2508.21148.

Metadata

| Field | Content | |-------|---------| | Title | A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers | | Authors | Ming Hu, Chenglong Ma, Wei Li, Wanghan Xu, Jiamin Wu, Jucheng Hu, et al. (120 authors in total) | | Published | October 2025 (arXiv) | | Source | https://arxiv.org/abs/2508.21148 | | Type | Survey |

Scope

The survey maps the landscape of large language models applied to scientific research, tracing the field from data foundations through model design to agentic systems. The forum post frames it within the broader context of agentic search and LLM-era information access.

Key points

• Unified perspective: The survey consolidates dispersed work on scientific LLMs into a comparable framework spanning data, models, and agents.
• Component decomposition: It breaks down method components (representation learning, retrievers, rerankers, planners, generators, feedback mechanisms) to support engineering adoption.
• Taxonomy: The post organizes approaches along modeling paradigm (discriminative vs. generative retrieval), LLM integration (RAG / agents / tool use), optimization objectives (relevance, diversity, freshness), and evaluation (offline, online, human assessment).
• Research timeline: BERT reranking and DPR established neural retrieval (2019–2021); RAG drove retrieval-generation fusion (2022–2023); conversational/agentic search and generative recommendation surged from 2024; RL-trained search agents, deep research, and GraphRAG are the 2025+ growth frontier.
• Open problems: benchmark credibility, latency and cost, hallucination and safety, cross-lingual and multimodal generalization, and reproducibility of private data.

Evaluation landscape discussed

• Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora, and public recommendation datasets.
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, and token cost.
• Baselines: BM25, dense retrieval, cross-encoder reranking, non-retrieval LLMs, commercial search APIs.

Insights for practitioners

1. Architecture: cascaded retrieve → rerank → generate remains mainstream, but agentic paradigms make retrieval policy itself learnable. 2. Data: high-quality instruction data and interaction logs are as critical as model design; synthetic data risks leakage and distribution shift. 3. Evaluation: offline metrics diverge from online satisfaction; LLM-as-judge should be cross-validated with human evaluation. 4. Deployment: latency, cost, interpretability, and safety are hard constraints in production, not just academic benchmarks.

Engineering checklist (from the post)

| Item | Question | Recommendation | |------|----------|----------------| | Data | PII handling, versioning | Partitioned indices, anonymization, rollback-safe embeddings | | Latency | p99 budget, retrieval depth | Cascades with early stopping, query caching, async reranking | | Quality | Does offline gain translate online? | Interleaving experiments, human audits, citation verification | | Safety | Poisoning/bias from open retrieval? | Source whitelists, adversarial detection, output filtering | | Cost | Token/GPU spend per query | Model routing, distillation, hybrid sparse-dense retrieval |

Related entries

• A Comprehensive Survey of Deep Research
• A Survey of LLM-based Deep Search Agents
• Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents
• Agentic Reasoning

Note

Much of the post's body uses a generic template; quantitative results should be verified against the original PDF at https://arxiv.org/abs/2508.21148.