Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents (Mar 2025, arXiv)

Overview

| Field | Content | |------|------| | Title | Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents | | Authors / Affiliations | Shuo Ren, Can Xie, Pu Jian, Zhenjiang Ren, Chunlin Leng, Jiajun Zhang | | Published | March 2025 | | Source | https://arxiv.org/abs/2503.24047 | | Type | Survey | | Section | Deep Research |

Background and Motivation

In large-scale search, recommendation, and personalization systems, agentic search has long faced challenges in efficiency, scalability, and user-intent understanding. Traditional pipeline approaches often split retrieval, ranking, and generation, making it hard to meet the LLM-era demand for natural-language interaction, multi-hop reasoning, and real-time knowledge. This survey was proposed against this backdrop to systematically map the theoretical and practical boundaries of LLM-based scientific agents — agents that plan, retrieve, use tools, and generate across scientific workflows.

Core Contributions

• Proposes or consolidates a unified perspective, bringing scattered work into a comparable framework.
• Clearly decomposes method components (representation learning, retrievers, rerankers, planners, generators, feedback mechanisms) for easier engineering adoption.
• Provides reproducible benchmarks, datasets, or taxonomy tables, lowering entry barriers for follow-up researchers.
• Discusses interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration, and the path from research prototypes to industrial systems.
• Lists open problems: evaluation trustworthiness, latency and cost, hallucination and safety, and cross-lingual / multimodal extension.

Taxonomy

| Dimension | Subtype | Representative ideas | Strengths | Limitations | |------|------|----------|------|----------| | Modeling paradigm | Discriminative / generative retrieval | Bi-encoder, cross-encoder, DSI, GPT indexing | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization objective | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Annotation scarcity | | Evaluation | Offline / Online / Human | nDCG, MRR, LLM-as-judge, A/B tests | Comparable | Divergence from real satisfaction |

Four Technical Lines

• Dense retrieval: high recall, low latency; suited to first-stage retrieval.
• Late interaction (e.g., ColBERT): higher accuracy but larger indexes.
• Generative IR: directly "generates" documents via tokens or docids, simplifying cascades.
• Agentic search: models search as sequential decision-making, supporting multi-hop reasoning and self-reflection.

Timeline of Research Evolution

• 2019–2021: BERT reranking and DPR lay the foundation of neural retrieval.
• 2022–2023: RAG and FreshLLMs drive retrieval-generation fusion.
• 2024: explosive growth in conversational/agentic search and Gen-RecSys.
• 2025–2026: RL-trained search agents, Deep Research, and GraphRAG become new growth poles.

Evaluation Paradigm

Typical benchmarks and trends covered:

• Datasets: MS MARCO, BEIR, Natural Questions, domain corpora, public recommendation datasets.
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency and token cost.
• Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs.
• Ablations: contribution of each module (retrieval steps, rerank depth, training data scale) to final quality.

Note: exact numerical results should be verified against the original PDF; this post is based on the abstract and public metadata.

Key Insights

1. Architecture: cascaded retrieve-rerank-generate remains mainstream, but the agentic paradigm makes "retrieval frequency and strategy" itself learnable. 2. Data: high-quality instruction data and click/session logs are equally critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human evaluation. 4. Products: latency, cost, interpretability, and safety are hard constraints for industrial deployment — do not optimize academic benchmarks alone.

Open Problems and Future Directions

Commonly cited gaps: insufficient unified benchmarks, irreproducible private data, LLM evaluation bias, and safety/cost constraints of agentic systems. Future work includes finer-grained process supervision, joint retrieval-reasoning training, enterprise metadata governance, and multimodal/cross-lingual consistency.

Limitations

Possible limitations include experiment scale constrained by GPU budgets, mismatch between benchmarks and real user distributions, English-centric data leaving cross-lingual generalization unknown, and safety risks of agents on the open web.

Related Entries

• A Comprehensive Survey of Deep Research
• A Survey of LLM-based Deep Search Agents
• A Survey of Scientific Large Language Models
• AgentIR: Reasoning-Aware Retrieval for Deep Research Agents
• Agentic Reasoning: A Streamlined Framework
• BioMedArena

References

• Original paper: Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents, arXiv:2503.24047.

Glossary

| Term | Meaning | |------|------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain | | Agentic Search | Modeling search as sequential decisions and tool calls | | Gen-IR | Generative Information Retrieval |