Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents (arXiv, March 2025)
Overview
- Paper: Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents
- Authors / Affiliations: Shuo Ren, Can Xie, Pu Jian, Zhenjiang Ren, Chunlin Leng, Jiajun Zhang
- Source: https://arxiv.org/abs/2503.24047
- Type: Survey
- Section: Deep Research
- A unified perspective that brings dispersed work on scientific agents into a comparable framework.
- A clear decomposition of method components: representation learning, retrievers, re-rankers, planners, generators, and feedback mechanisms.
- Reproducible benchmarks, datasets, and taxonomy tables that lower the entry barrier for follow-up researchers.
- Discussion of interfaces with LLM tool calling, reinforcement learning, and multi-agent collaboration, including migration paths from research prototypes to industrial systems.
- Explicit open problems: evaluation credibility, latency and cost, hallucination and safety, and cross-lingual / multimodal extension.
- Dense retrieval: high recall, low latency; suited to first-stage retrieval.
- Late interaction (e.g., ColBERT): higher precision but larger indexes.
- Generative IR: "generates" documents directly via tokens or docids, simplifying cascades.
- Agentic search: models search as sequential decision-making, supporting multi-hop reasoning and self-reflection.
- 2019–2021: BERT re-ranking and DPR lay the foundation of neural retrieval.
- 2022–2023: RAG and FreshLLM drive retrieval-generation fusion.
- 2024 onward: Conversational / agentic search and Gen-RecSys explode in popularity.
- 2025–2026: RL-trained search agents, Deep Research, and GraphRAG become new growth areas.
- Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora, public recommendation datasets.
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency and token cost.
- Baselines: BM25, dense retrieval, cross-encoder re-ranking, retrieval-free LLMs, commercial search APIs.
- A Comprehensive Survey of Deep Research: Systems, Methodologies, and A…
- A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Eval…
- A Survey of Scientific Large Language Models: From Data Foundations to…
- AgentIR: Reasoning-Aware Retrieval for Deep Research Agents, Mar 2026,…
- Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning…
- BioMedArena: An Open-source Toolkit for Building and Evaluating Biomed…
- Original paper: Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents, arXiv, March 2025. https://arxiv.org/abs/2503.24047
One-line summary
This survey systematically organizes research on LLM-based scientific agents, mapping how large language models can be orchestrated into autonomous systems that perform scientific research tasks end to end.
Background and motivation
Traditional research pipelines separate retrieval, ranking, and generation, which struggles to meet the demands of the LLM era: natural-language interaction, multi-hop reasoning, and up-to-date knowledge. LLM-based scientific agents emerged to address this gap — modeling research tasks as sequential decision-making with tool use, planning, and self-reflection — yet the field has lacked a unified taxonomy. This survey addresses that need by organizing scattered work into a comparable framework.
Core contributions
Taxonomy of approaches
| Dimension | Subcategories | Representative ideas | Strengths | Limitations | |---|---|---|---|---| | Modeling paradigm | Discriminative / generative retrieval | Bi-encoders, cross-encoders, DSI | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization objective | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Label scarcity | | Evaluation | Offline / Online / Human | nDCG, MRR, LLM-as-judge, A/B tests | Comparable | Deviates from real satisfaction |
Four main research lines are typically contrasted:
Timeline of research evolution
Evaluation paradigm
Common benchmarks and metrics covered by surveys in this space include:
> Note: exact quantitative results should be verified against the original PDF; this entry summarizes experimental design logic based on the abstract and public metadata.
Key insights
1. Architecture: cascaded retrieval + re-ranking + generation remains mainstream, but the agentic paradigm makes "how many times and how to retrieve" itself a learnable object. 2. Data: high-quality instruction data and click/session logs are equally critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation. 4. Products: latency, cost, interpretability, and safety are hard constraints for industrial deployment — not just academic benchmarks.
Open problems and future directions
Authors in this space commonly note insufficient unified benchmarks, non-reproducible private data, LLM evaluation bias, and the safety and cost constraints of agentic systems. Future work includes finer-grained process supervision, joint retrieval-reasoning training, enterprise metadata governance, and multimodal / cross-lingual consistency.
Limitations
Possible limitations include experiment scale bounded by GPU budgets, benchmark–real-user distribution mismatch, English-centric data leaving cross-lingual generalization unknown, and safety risks of agents operating on the open web.
Related entries in this list
Glossary
| Term | Meaning | |---|---| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking-quality metric | | Agentic Search | A paradigm modeling search as sequential decision-making and tool calling | | Gen-IR | Generative Information Retrieval |