PaSa: An LLM Agent for Comprehensive Academic Paper Search (arXiv 2501.10120, Jan 2025)

Forum topic · 小凯 · 2026-07-05

Summary

This forum post summarizes PaSa, an LLM-based agent for comprehensive academic paper search, published on arXiv in January 2025 (arXiv:2501.10120). The work is positioned at the intersection of agentic search and large-scale search/recommendation systems, addressing how to redistribute the responsibilities of retrieval, ranking, generation, and tool calling in the LLM era. The post provides a structured review covering the research background and motivation, core contributions, method/system architecture, experimental design, and open problems. It describes a four-step methodology (problem formulation, model/system design, training or construction, inference pipeline) and discusses components such as retrievers, rerankers, planners, memory modules, and tool interfaces, with learning strategies spanning supervised fine-tuning, contrastive learning, distillation, and reinforcement learning. The post also notes evaluation protocols using datasets like MS MARCO, BEIR, and Natural Questions, metrics such as nDCG@10, MRR, and Recall@k, and baselines including BM25 and dense retrieval. Additional sections offer an engineering checklist covering data privacy, latency budgets, output quality, security against corpus poisoning, and cost control, plus a glossary of IR/RAG terminology. Readers are advised to verify quantitative claims against the original paper PDF.

This post from zhichai.net discusses PaSa: An LLM Agent for Comprehensive Academic Paper Search (January 2025, arXiv).

Source link: https://arxiv.org/abs/2501.10120
Authors: Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, et al. (7 authors total)
Category: Verticals (agentic search)

Research Background and Motivation

The post situates PaSa in the context of agentic search for large-scale search, recommendation, and personalization systems. Traditional pipeline-style approaches separate retrieval, ranking, and generation, which struggles to meet the LLM-era demand for natural language interaction, multi-hop reasoning, and up-to-date knowledge. PaSa targets this gap, aiming to advance both the theory and practice of the intersection between agentic search and academic information access.

Core Contributions (as outlined in the post)

A unified perspective that organizes scattered related work into a comparable framework
A clear decomposition of method components: representation learning, retrievers, rerankers, planners, generators, and feedback mechanisms
Reproducible benchmarks, datasets, or taxonomy tables to lower the entry cost for follow-up researchers
Discussion of interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration
Open problems: evaluation trustworthiness, latency and cost, hallucination and safety, cross-lingual and multimodal extension

Method / System Architecture

The post describes a general four-step approach: problem formulation → model/system design → training/construction pipeline → inference pipeline:

1. Input and representation: encode queries, documents, and user context as dense or sparse representations, or structured prompts 2. Core modules: retrievers, rerankers, planners, memory modules, and tool interfaces, chained or run in parallel by task 3. Learning strategies: supervised fine-tuning, contrastive learning, distillation, reinforcement learning (including process rewards), and bootstrapped data synthesis 4. Inference strategies: single-round retrieval, iterative retrieval, parallel sub-queries, early stopping, and budget control

Evaluation

Typical experimental design discussed in the post:

Datasets: MS MARCO, BEIR, Natural Questions, domain corpora, public recommendation sets
Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency and token cost
Baselines: BM25, dense retrieval, cross-encoder reranking, non-retrieval LLMs, commercial search APIs

Note: the post explicitly advises readers to check the original PDF tables for exact numerical results, as the post is based on the abstract and public metadata.

Key Takeaways for Search / Rec / Personalization

1. Architecture: cascaded retrieval + reranking + generation remains mainstream, but the agentic paradigm makes retrieval count and strategy themselves learnable 2. Data: high-quality instruction data and click/session logs matter equally; synthetic data must guard against knowledge leakage and distribution shift 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human evaluation 4. Product: latency, cost, explainability, and safety are hard constraints for industrial deployment

Engineering Checklist

| Item | Question | Suggestion | |------|----------|------------| | Data | PII in training/index? Version control? | Partitioned indexes, anonymization, rollback-able embedding versions | | Latency | p99 budget? Retrieval steps? | Cascade + early stop, hot-query caching, async reranking | | Quality | Does offline gain translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Safety | Poisoning/bias from open retrieval? | Source whitelists, adversarial detection, output filtering | | Cost | Tokens and GPU per query? | Small-model routing, distillation, hybrid sparse+dense |

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain | | Agentic Search | Search modeled as sequential decision-making and tool calling | | Gen-IR | Generative Information Retrieval |

Related Entries

An interpretable ensemble of graph and language models
Applying Deep Learning to Ads Conversion Prediction in Last Mile Delivery (arXiv 2502.10514)
Automated Query-Product Relevance Labeling using LLMs (arXiv 2502.15990)
Behavior Modeling Space Reconstruction for E-Commerce Search (arXiv 2501.18216)
Behavior-driven query similarity prediction based on pre-trained language models
Better to Ask in English: Cross-Lingual Evaluation of LLMs (DOI: 10.1145/3589334.3645643)

Reference

Original paper: PaSa: An LLM Agent for Comprehensive Academic Paper Search. arXiv:2501.10120