Overview
RAG-Anything (arXiv:2510.12323v1, HKUDS) extends the LightRAG framework to a fully multimodal retrieval-augmented generation system for long, mixed-content documents such as research papers, financial reports, and medical records. The code is available at https://github.com/HKUDS/RAG-Anything.
The authors argue that text-only RAG pipelines lose critical information when documents contain charts, tables, formulas, and images. RAG-Anything is designed as an architectural extension of LightRAG, not a replacement, reusing LightRAG's hierarchical retrieval ideas while adding multimodal awareness.
Core Design
Atomic Content Units
Each document is decomposed into atomic units:
$$ c_j = (t_j, x_j) $$
where $t_j$ is the modality type (text, image, table, equation) and $x_j$ is the extracted raw content. The MinerU parser (arXiv:2409.18839) performs high-fidelity extraction that preserves multi-column layouts, nested tables, and inline formulas. Crucially, context anchoring is retained: each figure remains linked to its caption, each formula to its surrounding definitions, each table to explanatory text.
Dual-Graph Construction
RAG-Anything's central innovation is the construction of two complementary knowledge graphs and their fusion:
1. Cross-Modal Knowledge Graph. For each non-text unit, an MLLM produces a detailed description $d_j^{chunk}$ for retrieval and an entity summary $e_j^{entity}$ for graph construction. Generation is context-aware: each unit is processed with its local neighborhood $C_j = \{c_k \mid |k-j| \leq \delta\}$. A graph extraction routine then produces:
$$ (\mathcal{V}_j, \mathcal{E}_j) = R(d_j^{chunk}) $$
A multimodal entity node $v_j^{mm}$ anchors intra-chunk entities through explicit belongs_to edges.
2. Text-Based Knowledge Graph. For text units, the system reuses the LightRAG/GraphRAG pipeline of NER + RE.
3. Graph Fusion. The two graphs are merged via entity alignment using entity names as match keys, yielding a unified knowledge graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$.
Hybrid Retrieval
Query-time retrieval combines two signals:
- Structural knowledge navigation. Keywords extracted from the query trigger precise entity matches in $\mathcal{G}$ with controlled neighborhood expansion, enabling multi-hop cross-modal reasoning (e.g., from a paragraph to the figure it cites, then to a specific panel).
- Semantic similarity matching. The query embedding $\mathbf{e}_q$ is compared against an embedding table $\mathcal{T}$ containing atomic chunks, figure entities, and relation representations via dense vector similarity.
- Text components (entity summaries, relation descriptions, chunk text) form $\mathcal{P}(q)$.
- Visual components are dereferenced to recover original visual content $\mathcal{V}^*(q)$.
- GPT-4o-mini: 51.2%
- LightRAG: 58.4%
- MMGraphRAG: 61.0%
- RAG-Anything: 63.4%
- GPT-4o-mini: 33.5%
- LightRAG: 38.9%
- MMGraphRAG: 37.7%
- RAG-Anything: 42.8%
- DocBench >100 pages: RAG-Anything 68.2% vs MMGraphRAG 54.6% (+13.6)
- DocBench >200 pages: RAG-Anything 68.8% vs MMGraphRAG 55.0% (+13.8)
- MMLongBench 51-100 pages: +9.3
- MMLongBench 101-200 pages: +7.9
- LightRAG (arXiv:2410.05779): RAG-Anything inherits its text-graph construction and dual-level retrieval, then extends them.
- MMGraphRAG: handles only basic images; treats tables and formulas as plain text.
- VisRAG: preserves document layout as images but lacks fine-grained relational modeling.
- VideoRAG: targets video modalities; RAG-Anything targets document modalities. The two are complementary.
- MinerU for PDF, image, DOCX, PPTX, XLSX parsing.
- GPT-4o-mini as the MLLM for non-text unit description.
- text-embedding-3-large (3072-dim) embeddings.
- bge-reranker-v2-m3 reranker.
- Configurable graph-construction token limits (20,000 for entities/relations; 12,000 per chunk).
- Supports custom parser plugins, multilingual prompts, and local backends such as Ollama and vLLM.
- Compatible with LangChain and LlamaIndex ecosystems.
- How reliable is MLLM-generated description, and how do description hallucinations propagate through graph construction and retrieval?
- Is name-based entity alignment robust under abbreviations, synonyms, and cross-lingual cases?
- Does the added cost of dual-graph construction, per-unit MLLM calls, and cross-modal reranking scale to industrial deployment on very long documents?
- How well does the modality scheme (LaTeX for equations, structured cells for tables) generalize to 3D models, code blocks, or interactive charts?
- As VLM context windows grow (e.g., Gemini 1M+ tokens), will explicit graph structures retain their edge over direct end-to-end document ingestion?
- Guo et al. (2025). *RAG-Anything: All-in-One RAG Framework.* arXiv:2510.12323.
- Guo et al. (2024). *LightRAG: Simple and Fast Retrieval-Augmented Generation.* arXiv:2410.05779.
- Wang et al. (2024). *MinerU: An Open-Source Solution for Precise Document Content Extraction.* arXiv:2409.18839.
- Zou et al. (2024). *DocBench: A Benchmark for Evaluating LLM-based Document Reading Systems.* arXiv:2407.10701.
- Ma et al. (2024). *MMLongBench-Doc: Benchmarking Long-Context Document Understanding with Visualizations.* NeurIPS 37.
Candidates are merged as $\mathcal{C}(q) = \mathcal{C}_{stru}(q) \cup \mathcal{C}_{seman}(q)$ and reranked using a multi-signal fusion score that weights structural importance, semantic similarity, and inferred modality preference from the query.
Generation
Retrieved candidates are partitioned:
A VLM then conditions jointly on $q$, $\mathcal{P}(q)$, and $\mathcal{V}^*(q)$ to produce the final response.
Experimental Results
Benchmarks
| Dataset | Documents | Avg. Pages | Avg. Tokens | Questions | |---|---|---|---|---| | DocBench | 229 | 66 | 46,377 | 1,102 | | MMLongBench | 135 | 47.5 | 21,214 | 1,082 |
Main Results (Overall Accuracy)
DocBench:
MMLongBench:
Long-Document Effect
The advantage grows with document length:
Ablation (DocBench)
| Variant | Overall | |---|---| | Chunk-only (no graph) | 60.0% | | w/o Reranker | 62.4% | | Full RAG-Anything | 63.4% |
Graph construction accounts for the largest contribution; reranking adds a smaller incremental gain.
Qualitative Cases
Multi-panel figures. In a t-SNE visualization with separate style-space and content-space panels, the layout graph encodes panel boundaries, axis titles, legends, and captions as nodes with edges such as panel_contains_plot, caption_provides_context, and subfigure_relates_hierarchically, guiding retrieval to the correct panel.
Financial table navigation. A query for the intersection of "Wages and salaries" and "2020" in a multi-period table (26,778 million) succeeds because row headers, column headers, data cells, and units are all graph nodes with row-of, column-of, header-applies-to, and unit-of edges, enabling precise cell-level navigation.
Related Work
Limitations
The authors note two systemic issues:
1. Text-centric retrieval bias. Even when queries explicitly request visual information, the system tends to favor textual sources. 2. Complex layout handling. Merged cells and ambiguous table boundaries still cause errors.
Both reflect fundamental challenges in cross-modal attention and layout-aware parsing.