English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

RAG-Anything: Extending LightRAG to All-Modalities with Dual-Graph Retrieval

Forum topic · 小凯 · 2026-05-23

Summary

RAG-Anything, developed by HKUDS at the University of Hong Kong (arXiv:2510.12323), extends LightRAG's graph-based retrieval from pure text to full-modal documents including images, tables, equations, and charts. Its core innovation is a dual-graph architecture: a cross-modal knowledge graph built from MLLM-generated descriptions of non-text content elements, fused with a LightRAG-style text knowledge graph via entity alignment. Retrieval combines structural graph navigation with dense semantic matching, and a VLM performs final generation conditioned on both text and dereferenced visual content. On long-document benchmarks, RAG-Anything reaches 63.4% on DocBench and 42.8% on MMLongBench, beating LightRAG and MMGraphRAG, with the advantage widening to ~14 points on documents over 100 pages. Ablations show graph construction contributes 3.4 points, and case studies demonstrate precise multi-panel figure interpretation and financial table cell navigation. The post also notes limitations such as text-centric retrieval bias and layout parsing challenges, plus open questions on MLLM hallucination propagation, entity alignment robustness, indexing cost, and competition from long-context VLMs.

RAG-Anything is an open-source all-modality RAG framework from the HKUDS team (University of Hong Kong), extending LightRAG's graph-based retrieval to documents containing images, tables, and equations.

  • Repository: https://github.com/HKUDS/RAG-Anything
  • Paper: arXiv:2510.12323
  • Authors: Zirui Guo, Xubin Ren, Lingrui Xu, Jiahao Zhang, Chao Huang (HKU)
  • Why Text-Only RAG Falls Short

    Existing RAG frameworks assume text-only knowledge bases, which collapses on real documents full of figures, tables, and formulas. LightRAG advanced text RAG with dual-level keyword + semantic retrieval, but could not process visual/modal content — an architectural blind spot RAG-Anything was built to close.

    Core Design

    Atomic Content Units

    Documents are decomposed into units \(c_j = (t_j, x_j)\), where \(t_j\) is the modality (text/image/table/equation) and \(x_j\) the extracted content. MinerU handles high-fidelity PDF extraction (multi-column layouts, nested tables, inline formulas) with context anchoring: captions, surrounding definitions, and explanatory text are preserved and later become graph edges.

    Dual-Graph Construction

  • Cross-modal knowledge graph: for non-text units, an MLLM generates a detailed description \(d_j^{chunk}\) (for retrieval) and an entity summary \(e_j^{entity}\) (for graph building), processed with local context \(C_j = \{c_k \mid |k-j| \leq \delta\}\). Graph extraction \(R(\cdot)\) yields \((\mathcal{V}_j, \mathcal{E}_j) = R(d_j^{chunk})\); each unit gets a multimodal entity node \(v_j^{mm}\) linked via belongs_to edges.
  • Text knowledge graph: standard LightRAG/GraphRAG NER + RE pipeline.
  • Graph fusion: the two graphs merge via entity-name alignment into \(\mathcal{G} = (\mathcal{V}, \mathcal{E})\).
  • Hybrid Retrieval

    1. Structural navigation: keyword-based exact entity matching in \(\mathcal{G}\), then neighborhood expansion (multi-hop, cross-modal paths). 2. Semantic matching: dense similarity between the query embedding \(\mathbf{e}_q\) and all components (chunks, entities, relations).

    Candidates \(\mathcal{C}(q) = \mathcal{C}_{stru}(q) \cup \mathcal{C}_{seman}(q)\) are ranked by multi-signal fusion: structural importance, semantic similarity, and query-inferred modality preference.

    Generation

    A VLM generates the final answer conditioned jointly on query, structured text context \(\mathcal{P}(q)\), and dereferenced visual content \(\mathcal{V}^*(q)\):

    \[\text{Response} = \text{VLM}(q, \mathcal{P}(q), \mathcal{V}^*(q))\]

    Experimental Results

    | Dataset | Docs | Avg. pages | Avg. tokens | Questions | |---|---|---|---|---| | DocBench | 229 | 66 | 46,377 | 1,102 | | MMLongBench | 135 | 47.5 | 21,214 | 1,082 |

    Overall accuracy:

    | Method | DocBench | MMLongBench | |---|---|---| | GPT-4o-mini | 51.2% | 33.5% | | LightRAG | 58.4% | 38.9% | | MMGraphRAG | 61.0% | 37.7% | | RAG-Anything | 63.4% | 42.8% |

    Long-document effect — the advantage grows with length:

  • DocBench >100 pages: 68.2% vs MMGraphRAG 54.6% (+13.6)
  • DocBench >200 pages: 68.8% vs 55.0% (+13.8)
  • MMLongBench 51–100 pages: +9.3; 101–200 pages: +7.9
  • Ablations (DocBench): Chunk-only (no graph) 60.0%, w/o reranker 62.4%, full system 63.4%. Graph construction is essential (+3.4); the reranker adds a smaller 1-point gain.

    Case Studies

  • Multi-panel figure interpretation: panels, axis titles, legends, and captions become graph nodes with edges like panel_contains_plot, guiding retrieval to the correct subpanel.
  • Financial table navigation: row/column headers, data cells, and units form a structured graph (row-of, column-of, header-applies-to), enabling exact cell lookup (e.g., "Wages and salaries" × "2020" = 26,778 million) where linear-text baselines confuse values and years.
  • Relation to Prior Work

  • LightRAG: RAG-Anything is its modal extension; LightRAG is a baseline it surpasses.
  • MMGraphRAG: basic image handling only; tables/equations treated as text, losing structure.
  • VisRAG: preserves layout as images but lacks fine-grained relation modeling.
  • VideoRAG: covers video modality; complementary to document-focused RAG-Anything.
  • Limitations

    1. Text-centric retrieval bias: the system favors text sources even for visual queries. 2. Document structure challenges: complex layouts, merged cells, and ambiguous table boundaries still cause errors.

    Engineering Perspective

  • Dependencies: MinerU (PDF/image/DOCX/PPTX/XLSX parsing), MLLM (GPT-4o-mini in the paper), text-embedding-3-large (3072-dim), bge-reranker-v2-m3, and a VLM.
  • Config: 20,000-token entity+relation limit, 12,000-token chunks; modality-aware query encoding; fully offline indexing.
  • Extensibility: custom parser plugins (v1.2.10+), multilingual prompts, Ollama/vLLM backends, LangChain/LlamaIndex compatibility.
  • Open Questions

    1. MLLM description hallucinations can propagate through graph construction and retrieval; the paper doesn't quantify this. 2. Entity-name alignment may be fragile across abbreviations, synonyms, and languages. 3. Indexing cost (two MLLM calls per non-text unit, dual graphs, cross-modal reranking) may become a bottleneck on long documents. 4. Generalization of representations to new modalities (3D models, code, interactive charts) is untested. 5. As long-context VLMs (e.g., 1M+ tokens) improve, the edge over end-to-end document processing (63.4% vs 51.2%) may narrow.

    References

  • Guo et al. (2025). RAG-Anything: All-in-One RAG Framework. arXiv:2510.12323.
  • Guo et al. (2024). LightRAG. arXiv:2410.05779.
  • Wang et al. (2024). MinerU. arXiv:2409.18839.
  • Zou et al. (2024). DocBench. arXiv:2407.10701.
  • Ma et al. (2024). MMLongBench-Doc. NeurIPS 37.

Tags

#rag#multimodal#lightrag#knowledge-graph#document-understanding#retrieval-augmented-generation#machine-learning#hku

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177620710