RAG vs. GraphRAG: A Systematic Evaluation and Key Insights (arXiv 2502.11371)
Source: https://arxiv.org/abs/2502.11371 — February 2025, arXiv. Authors include Haoyu Han, Li Ma, Yu Wang, Harry Shomer, Yongjia Lei, Zhisheng Qi, and others (12 authors total).
Key points
- The paper provides a systematic comparison between RAG (Retrieval-Augmented Generation) and GraphRAG (Graph Retrieval-Augmented Generation), examining how graph-structured retrieval differs from conventional chunk-based retrieval when augmenting LLMs.
- Context: RAG pipelines have long faced challenges in efficiency, scalability, and user-intent understanding; traditional pipelines separate retrieval, ranking, and generation, which struggles with multi-hop reasoning and real-time knowledge needs in the LLM era.
- The post situates the work in a broader research timeline:
- 2019–2021: BERT rerankers and DPR establish neural retrieval foundations.
- 2022–2023: RAG and FreshLLMs drive retrieval-generation fusion.
- 2024: Conversational/agentic search and generative recommender systems (Gen-RecSys) proliferate.
- 2025–2026: RL-trained search agents, Deep Research, and GraphRAG become new growth areas.
- Common datasets: MS MARCO, BEIR, Natural Questions, domain corpora.
- Common metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency and token cost.
- Typical baselines: BM25, dense retrieval, cross-encoder rerankers, no-retrieval LLMs, commercial search APIs.
- Note: quantitative results should be verified against the original PDF.
- A Survey of Graph Retrieval-Augmented Generation for Customized LLMs
- Agentic RAG: A Survey on Agentic Retrieval-Augmented Generation
- RAFT: Adapting Language Models to Domain-Specific RAG (2024)
Taxonomy discussed
| Dimension | Subtypes | Representative approaches | Strengths | Limitations | |---|---|---|---|---| | Retrieval paradigm | Dense / generative retrieval | Dual-tower, cross-encoders, DSI | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization goals | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Scarce annotations | | Evaluation | Offline / Online / Human | nDCG, MRR, LLM-as-judge, A/B tests | Comparable | Diverges from real satisfaction |
Four main retrieval lines are contrasted: dense retrieval (high recall, low latency), late interaction (e.g., ColBERT — higher precision, larger indexes), generative IR (directly generating documents/docids), and agentic search (search as sequential decision-making with multi-hop and self-reflection).
Evaluation paradigm
Main insights
1. Architecture: Cascaded retrieval + reranking + generation remains mainstream, but agentic paradigms make retrieval count and strategy themselves learnable. 2. Data: High-quality instruction data and click/session logs are critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human evaluation. 4. Product: Latency, cost, explainability, and safety are hard industrial constraints — do not optimize only academic benchmarks.
Open problems
Insufficient unified benchmarks, non-reproducible private data, LLM evaluation bias, and safety/cost constraints for agentic systems. Future directions include finer-grained process supervision, retrieval-reasoning co-training, enterprise metadata governance, and multimodal/cross-lingual consistency.
Engineering checklist (from the post)
| Check | Question | Recommendation | |---|---|---| | Data | PII in index? Versioning? | Partitioned indexes, anonymization, rollback-able embeddings | | Latency | p99 budget? Retrieval steps? | Cascade + early exit, query caching, async reranking | | Quality | Offline gains → online CTR? | Interleaving experiments, human audits, citation checks | | Safety | Poisoning/bias via open retrieval? | Source whitelists, adversarial detection, output filtering | | Cost | Token/GPU cost per query? | Small-model routing, distillation, hybrid sparse+dense |