Graphify Deep Dive: Giving Karpathy's /raw Folder a Brain

Forum topic · 小凯 · 2026-04-07

Summary

Graphify is an open-source tool that converts unstructured file collections—code, PDFs, screenshots, whiteboard photos, Markdown notes—into a persistent, queryable knowledge graph for AI coding assistants. Motivated by Andrej Karpathy's well-known '/raw' folder problem, where an agent reads 200k+ tokens of files per query, Graphify achieves a reported 71.5x token compression by extracting knowledge once and traversing the graph on demand. It uses a dual-channel extraction engine: deterministic AST parsing via tree-sitter for 15+ languages (capturing imports, call graphs, and WHY/HACK/NOTE rationale comments), and LLM-based semantic extraction for documents and images via Claude sub-agents. Every edge carries a three-level confidence label (EXTRACTED, INFERRED, AMBIGUOUS) for honesty. Leiden community detection runs purely on graph topology with no vector database. Outputs include interactive HTML visualizations, audit reports (god nodes, surprising connections), Obsidian vaults, Neo4j scripts, and an MCP server. It integrates with Claude Code, Codex, OpenCode, and OpenClaw, supports Git hooks, file watching, SHA256-based incremental updates, and runs fully locally for code files. MIT licensed, ~2.2k GitHub stars.

Overview

Graphify addresses a problem Andrej Karpathy publicly complained about: AI coding assistants re-reading entire unstructured folders (his famous /raw folder—52 files, 200k+ tokens, $2-3 per query) just to answer simple questions. Graphify's core idea: don't treat raw files as the knowledge base—treat a structured knowledge graph as the knowledge base.

Key points

71.5x token compression: Karpathy's 52-file / 200k+ token material pack becomes graph queries of ~2.8k tokens. Build once, reuse across sessions; incremental updates only process changed files.
Dual-channel extraction engine:
Channel A — deterministic AST extraction (zero LLM cost): tree-sitter parsing for Python, TypeScript, JavaScript, Go, Rust, Java, C, C++, Ruby, C#, Kotlin, Scala, PHP, Swift, Lua, Zig, PowerShell, Elixir, Objective-C. Extracts classes, function signatures, imports/exports, call graphs, docstrings, and special comment markers (# WHY:, # HACK:, # NOTE:, # IMPORTANT:) as rationale_for edges—capturing not just what code does but why.
Channel B — semantic extraction (LLM sub-agents): parallel Claude sub-agents process Markdown/text, PDFs (citation mining), and images via Claude Vision. A whiteboard photo becomes an architecture node; a paper's concepts link to code classes—enabling a multimodal knowledge graph.
Three-level confidence labels ("honest AI"):
EXTRACTED (confidence 1.0): explicit relationships found in source.
INFERRED (0.6–0.9): plausible inferences, e.g., semantic similarity between functions.
AMBIGUOUS: flagged for human review.
Graph-topology community detection: Leiden algorithm (via graspologic) clusters purely on edge density—no embeddings, no vector database. The analysis surfaces God Nodes (highest-degree hubs like a BaseHandler depended on by 15 modules), Surprising Connections (cross-community, cross-modality links, with Code-Paper edges weighted above Code-Code), and Suggested Questions.
Unix-philosophy pipeline: detect() → extract() → build_graph() → cluster() → analyze() → report() → export(), each a single side-effect-free function communicating via plain dicts and NetworkX graphs.
Outputs (in graphify-out/):
graph.html — interactive vis.js visualization
GRAPH_REPORT.md — audit report with god nodes, surprising connections, suggested questions
graph.json — persistent graph data
cache/ — SHA256 cache for incremental updates
Optional flags: --obsidian (bidirectionally linked vault), --wiki, --svg, --graphml (Gephi/yEd), --neo4j / --neo4j-push, --mcp (MCP server for real-time agent queries)
Platform integrations: Claude Code, Codex, OpenCode, and OpenClaw. On OpenClaw, semantic extraction runs sequentially (multi-agent support is early-stage), and always-on behavior works only via AGENTS.md (no PreToolUse hooks). Query commands include /graphify query, /graphify path (shortest path), and /graphify explain.
Operations automation: Git hooks (post-commit/post-checkout AST-only rebuilds with zero LLM cost), --watch file monitoring with 3-second debounce, SHA256-based --update incremental merging, and a feedback loop where query results are saved to graphify-out/memory/ and folded back into the graph.
Typical use cases: onboarding onto an unfamiliar codebase in minutes; building a personal multimodal knowledge vault (papers, tweets, whiteboard photos, code snippets) with Obsidian export; tracing dependency paths in code review (with inferred edges clearly labeled); ingesting external papers via graphify add https://arxiv.org/abs/1706.03762 --author "Vaswani".
Stack & privacy: NetworkX, graspologic (Leiden), tree-sitter, Claude API, vis.js. No Neo4j, no server, fully local. Code files never leave the machine (local tree-sitter); only documents/images go to the Anthropic API with your own key. No telemetry or tracking.

References

GitHub: https://github.com/safishamsi/graphify
Install: pip install graphifyy && graphify install
Tutorial: https://www.aivi.fyi/llms/graphify
License: MIT (~2.2k stars)