RAG-Anything is an open-source all-modality RAG framework from the HKUDS team (University of Hong Kong), extending LightRAG's graph-based retrieval to documents containing images, tables, and equations.
- Repository: https://github.com/HKUDS/RAG-Anything
- Paper: arXiv:2510.12323
- Authors: Zirui Guo, Xubin Ren, Lingrui Xu, Jiahao Zhang, Chao Huang (HKU)
- Cross-modal knowledge graph: for non-text units, an MLLM generates a detailed description \(d_j^{chunk}\) (for retrieval) and an entity summary \(e_j^{entity}\) (for graph building), processed with local context \(C_j = \{c_k \mid |k-j| \leq \delta\}\). Graph extraction \(R(\cdot)\) yields \((\mathcal{V}_j, \mathcal{E}_j) = R(d_j^{chunk})\); each unit gets a multimodal entity node \(v_j^{mm}\) linked via
belongs_toedges. - Text knowledge graph: standard LightRAG/GraphRAG NER + RE pipeline.
- Graph fusion: the two graphs merge via entity-name alignment into \(\mathcal{G} = (\mathcal{V}, \mathcal{E})\).
- DocBench >100 pages: 68.2% vs MMGraphRAG 54.6% (+13.6)
- DocBench >200 pages: 68.8% vs 55.0% (+13.8)
- MMLongBench 51–100 pages: +9.3; 101–200 pages: +7.9
- Multi-panel figure interpretation: panels, axis titles, legends, and captions become graph nodes with edges like
panel_contains_plot, guiding retrieval to the correct subpanel. - Financial table navigation: row/column headers, data cells, and units form a structured graph (
row-of,column-of,header-applies-to), enabling exact cell lookup (e.g., "Wages and salaries" × "2020" = 26,778 million) where linear-text baselines confuse values and years. - LightRAG: RAG-Anything is its modal extension; LightRAG is a baseline it surpasses.
- MMGraphRAG: basic image handling only; tables/equations treated as text, losing structure.
- VisRAG: preserves layout as images but lacks fine-grained relation modeling.
- VideoRAG: covers video modality; complementary to document-focused RAG-Anything.
- Dependencies: MinerU (PDF/image/DOCX/PPTX/XLSX parsing), MLLM (GPT-4o-mini in the paper), text-embedding-3-large (3072-dim), bge-reranker-v2-m3, and a VLM.
- Config: 20,000-token entity+relation limit, 12,000-token chunks; modality-aware query encoding; fully offline indexing.
- Extensibility: custom parser plugins (v1.2.10+), multilingual prompts, Ollama/vLLM backends, LangChain/LlamaIndex compatibility.
- Guo et al. (2025). RAG-Anything: All-in-One RAG Framework. arXiv:2510.12323.
- Guo et al. (2024). LightRAG. arXiv:2410.05779.
- Wang et al. (2024). MinerU. arXiv:2409.18839.
- Zou et al. (2024). DocBench. arXiv:2407.10701.
- Ma et al. (2024). MMLongBench-Doc. NeurIPS 37.
Why Text-Only RAG Falls Short
Existing RAG frameworks assume text-only knowledge bases, which collapses on real documents full of figures, tables, and formulas. LightRAG advanced text RAG with dual-level keyword + semantic retrieval, but could not process visual/modal content — an architectural blind spot RAG-Anything was built to close.
Core Design
Atomic Content Units
Documents are decomposed into units \(c_j = (t_j, x_j)\), where \(t_j\) is the modality (text/image/table/equation) and \(x_j\) the extracted content. MinerU handles high-fidelity PDF extraction (multi-column layouts, nested tables, inline formulas) with context anchoring: captions, surrounding definitions, and explanatory text are preserved and later become graph edges.Dual-Graph Construction
Hybrid Retrieval
1. Structural navigation: keyword-based exact entity matching in \(\mathcal{G}\), then neighborhood expansion (multi-hop, cross-modal paths). 2. Semantic matching: dense similarity between the query embedding \(\mathbf{e}_q\) and all components (chunks, entities, relations).Candidates \(\mathcal{C}(q) = \mathcal{C}_{stru}(q) \cup \mathcal{C}_{seman}(q)\) are ranked by multi-signal fusion: structural importance, semantic similarity, and query-inferred modality preference.
Generation
A VLM generates the final answer conditioned jointly on query, structured text context \(\mathcal{P}(q)\), and dereferenced visual content \(\mathcal{V}^*(q)\):Experimental Results
| Dataset | Docs | Avg. pages | Avg. tokens | Questions | |---|---|---|---|---| | DocBench | 229 | 66 | 46,377 | 1,102 | | MMLongBench | 135 | 47.5 | 21,214 | 1,082 |
Overall accuracy:
| Method | DocBench | MMLongBench | |---|---|---| | GPT-4o-mini | 51.2% | 33.5% | | LightRAG | 58.4% | 38.9% | | MMGraphRAG | 61.0% | 37.7% | | RAG-Anything | 63.4% | 42.8% |
Long-document effect — the advantage grows with length:
Ablations (DocBench): Chunk-only (no graph) 60.0%, w/o reranker 62.4%, full system 63.4%. Graph construction is essential (+3.4); the reranker adds a smaller 1-point gain.
Case Studies
Relation to Prior Work
Limitations
1. Text-centric retrieval bias: the system favors text sources even for visual queries. 2. Document structure challenges: complex layouts, merged cells, and ambiguous table boundaries still cause errors.Engineering Perspective
Open Questions
1. MLLM description hallucinations can propagate through graph construction and retrieval; the paper doesn't quantify this. 2. Entity-name alignment may be fragile across abbreviations, synonyms, and languages. 3. Indexing cost (two MLLM calls per non-text unit, dual graphs, cross-modal reranking) may become a bottleneck on long documents. 4. Generalization of representations to new modalities (3D models, code, interactive charts) is untested. 5. As long-context VLMs (e.g., 1M+ tokens) improve, the edge over end-to-end document processing (63.4% vs 51.2%) may narrow.References