PageIndex: From Similarity Retrieval to Reasoning-Based Retrieval
> Project: PageIndex (GitHub: VectifyAI/PageIndex, ~30k stars) > Team: Mingtian Zhang, Yu Tang, VectifyAI Team > License: MIT > Core claim: Similarity ≠ Relevance — retrieval needs reasoning, not vector similarity > Benchmark: Mafin 2.5 financial RAG → 98.7% on FinanceBench
Key Points
- The similarity trap: Traditional vector RAG assumes semantic similarity equals relevance. For financial documents this fails — a chunk resembling "Net Income" may come from the wrong company's statement, chunking shreds table headers, and vector space ignores document structure.
- Core idea: PageIndex automates how human experts read long documents — check the table of contents, reason about which chapter likely contains the answer, then narrow to the section and page.
- Pipeline:
Parse → Detect TOC → Build Tree → Verify → Enrich → Search, using PyMuPDF for parsing, pattern matching + LLM assistance for TOC detection, and LLM summaries for node enrichment. - Three tree-building modes: TOC with page numbers, TOC without page numbers (LLM infers page mapping), and no TOC (chunk, generate initial tree, expand incrementally).
- Self-healing verification: sampled section titles are checked via LLM against their assigned pages; accuracy <60% triggers degradation to a simpler mode, 60–100% triggers up to 3 repair retries.
- The 31% vs 98.7% gap reflects a retrieval architecture gap, not an LLM capability gap — web/vector search can't precisely locate tables, footnotes, and notes in filings.
- Mafin 2.5 scored 98.7% with both GPT-4o and DeepSeek v3, implying the retrieval framework contributes more than the underlying model.
- Caveat: FinanceBench question design and data distribution aren't fully public, so independent verification of overfitting is impossible.
- Latency/cost: multiple LLM calls per query and for indexing; possible mitigations include tree caching and small-model pruning with large-model final judgment.
- TOC-less documents: auto-generated trees are of uncertain quality.
- PDF parsing: PyMuPDF-based; complex layouts still error-prone — vision models could help.
- Not applicable to code repos, chat logs, or unstructured short text, where vector RAG remains efficient.
- Single-document trees: cross-document queries need inter-document graphs and global indexes.
- PageIndex GitHub: https://github.com/VectifyAI/PageIndex
- Mafin 2.5 FinanceBench results: https://github.com/VectifyAI/Mafin2.5-FinanceBench
- TypeScript port: https://github.com/FutureSpeakAI/agent-fridays-pageindex-rag
- VectifyAI: https://vectify.ai
- FinanceBench paper: https://arxiv.org/abs/2311.11944 (Patel et al., 2023)
How Retrieval Works (Agentic RAG)
1. get_document_structure(doc_id) returns the tree structure (headings + summaries, no full text — saves tokens).
2. The LLM reasons over the tree: which branch most likely contains the answer?
3. get_page_content(doc_id, "21-28") extracts only the target pages.
4. The LLM answers from those exact pages, enabling page-level citations and auditable retrieval paths.
FinanceBench Results
| System | Accuracy | Full benchmark | Public results | |---|---|---|---| | Mafin 2.5 (PageIndex) | 98.7% | 100% | Yes | | Quantly | 94% | 100% | No | | Fintool | 98% | 66.7% | No | | ChatGPT 4o + Search | 31% | 66.7% | No | | Perplexity | 45% | 66.7% | No |
Vector RAG vs PageIndex
| Dimension | Vector RAG | PageIndex | |---|---|---| | Retrieval | Vector similarity | LLM reasoning over tree | | Storage | Vector DB (Pinecone/Weaviate/Chroma) | No vector DB; plain JSON index | | Explainability | Black-box similarity | Transparent ("Risk Factors → p21-28") | | Latency | Milliseconds | Seconds (multiple LLM calls) | | Token cost | Near zero at retrieval | LLM calls for indexing + retrieval | | Citations | Usually none | Exact page and section | | Best for | Short text, general knowledge | Long structured docs (finance/legal/academic) |
The AlphaGo analogy: PageIndex mimics policy-guided tree search — the LLM acts as the policy network deciding which branch to explore, but via explicit, interpretable reasoning rather than a trained network. "No Vector DB. No Chunking" reduces system complexity and vendor lock-in, but trades latency and LLM dependency.
Limitations and Outlook
Conclusion
PageIndex's value is not just the 98.7% figure but a paradigm proposal: documents are trees, not chunk clouds; retrieval is reasoned navigation, not nearest-neighbor matching. It is not a death sentence for vector databases — rather, structured knowledge needs structured retrieval. The likely endgame is a hybrid architecture: tree-based retrieval for structured long documents, vector retrieval for short/unstructured text, with query routing between the two.