HydraDB: A Graph Database That Lives in S3, with Disposable Compute Nodes
> GitHub: hydra-db/hydradb | Rust 1.91+ | AGPL-3.0 | Passes the OpenCypher TCK
Over the past few years, databases have shown a clear trend: storage-compute disaggregation. Snowflake popularized the idea — data lives in S3, compute nodes spin up on demand and get thrown away afterward. But this idea never really penetrated the graph database world. Neo4j is storage-compute coupled, and so is TigerGraph. Graph databases seem to inherently require "data on local SSD for fast traversal."
HydraDB aims to prove that a graph database can also decouple storage and compute — without sacrificing query performance.
Why graph traversal "needs" local storage
First, why has no one done storage-compute separation for graphs?
Relational queries "scan a batch of rows" — mostly sequential reads, well within S3's throughput. Graph queries "start from one node and hop N hops along edges" — every hop is a random access. S3 random access latency is on the order of 10–50ms, while local NVMe is around 0.1ms. A two-orders-of-magnitude difference.
This means if graph data lives in S3, every traversal gets throttled by S3 latency. That's why Neo4j requires data on local disk — not a design preference, but a performance requirement.
HydraDB's solution is pre-built immutable traversal indexes. Instead of doing random access at query time, it builds sparse-matrix traversal indexes (CSC format) at write time and stores them in S3. At query time, compute nodes pull the indexes to a local SSD cache and do traversal locally — at speeds close to a native graph database.
Two roles: Data Node and Indexer
HydraDB's architecture has two independent compute roles:
Data Node (graph-node) handles queries and writes. It reads data blocks and indexes from S3 into a local cache, and executes queries against that cache. A Data Node's local state is "disposable" — shutting one down loses no data, because S3 is the source of truth. On restart, the cache is rebuilt from S3.
Indexer (graph-indexer) builds traversal indexes in the background. After new data is written, the Indexer compiles the latest graph structure into an immutable CSC matrix and pushes it to S3. At query time, a Data Node reads both the "compiled index" and the "not-yet-compiled WAL increments," merging them for consistent results.
The key to this design: the Indexer is also disposable. If an Indexer dies, queries are unaffected — only newly written data is temporarily not compiled into the index. On restart, the Indexer resumes from its last position.
The two roles scale independently: more Data Nodes for query-heavy periods, more Indexers for write-heavy periods. S3 holds a single copy of the data.
SlateDB: LSM-tree on object storage
Under the hood, HydraDB uses SlateDB — a storage engine that runs an LSM-tree on object storage. A traditional LSM-tree keeps its WAL and SSTables on local disk; SlateDB puts them in S3.
This yields one direct benefit — write durability no longer depends on local disks. After a Data Node receives a write, the WAL goes straight to S3, with only the in-memory MemTable kept locally. If a Data Node crashes, unflushed data is in the S3 WAL and is not lost.
SlateDB's writer epoch mechanism solves the "multiple Data Nodes writing simultaneously" problem — via CAS (Compare-And-Swap) leases on object storage, one active writer is elected and old writers are fenced from writing further. It resembles Spanner's Paxos group mechanism, but runs on S3 rather than a distributed consensus protocol.
GraphBLAS: turning graph traversal into linear algebra
HydraDB's traversal engine uses SuiteSparse GraphBLAS — expressing graph traversal as sparse matrix operations.
The essence of graph traversal is matrix multiplication: adjacency matrix A multiplied by vector v (the starting-node vector) gives "one-hop neighbors." Multiply again for "two-hop neighbors." GraphBLAS turns this mathematical fact into engineering: instead of following edge pointers, traversal is done via matrix multiplication.
The benefit: matrix multiplication has mature optimized libraries (SIMD, GPU acceleration) and is far faster than pointer chasing. Moreover, GraphBLAS sparse formats (CSC/CSR) naturally suit compressed storage in S3 — sparse graphs produce sparse matrices with high compression ratios.
HydraDB's query planner picks strategies based on query patterns: property indexes, reverse adjacency, sparse traversal, GraphBLAS matrix operations — combined as needed.
OpenCypher and Bolt protocol compatibility
HydraDB supports two query interfaces:
- OpenCypher: Neo4j's query language standard, passing the OpenCypher TCK (Technology Compatibility Kit). Existing Cypher queries can run on HydraDB directly.
- Bolt protocol: Neo4j's binary protocol, version 5.x. Existing Neo4j drivers can connect to HydraDB directly.
- Neo4j: storage-compute coupled; single machine or causal cluster. Best for small-to-medium graphs; strongest local performance.
- TigerGraph: MPP architecture; distributed but storage-compute coupled. For large graphs; operationally complex.
- Memgraph: in-memory graph database; sub-millisecond latency but capacity limited by RAM.
- HydraDB: storage-compute separated, S3-persisted, disposable compute nodes. For cloud-native deployment and elastic scaling.
- Heavy graph database users: if your graph is growing and the cost of operating a Neo4j cluster keeps rising, HydraDB's disaggregated model is worth evaluating.
- Cloud-native architects: HydraDB is the graph-domain practice of the "storage-compute disaggregation" idea; its architecture (Data Node + Indexer + S3) is worth studying.
- Rust infrastructure developers: HydraDB is written in Rust 1.91+; the SlateDB + GraphBLAS combination is another case study of Rust in infrastructure.
- Teams needing compliance/audit: data in S3 naturally supports versioning (S3 Object Versioning), so audit requirements can be met with native object-storage capabilities.
The value of compatibility is migration cost — an application running Neo4j can point at HydraDB by changing a connection string, without touching queries. For teams migrating from Neo4j to cloud-native architecture, this is the lowest-cost path.
Positioning versus other graph databases
HydraDB's positioning is not "faster than Neo4j" — Neo4j is still faster in single-machine scenarios. HydraDB plays in a different dimension: when your graph is too big for one machine and query load has obvious peaks and valleys, the elasticity of storage-compute separation matters more than single-node speed.
A 10-billion-edge graph requires Neo4j to run an expensive cluster, and scaling means moving data. HydraDB keeps it in S3, spins up Data Nodes on demand, and shuts them down when idle — you only pay for S3 storage.
Who should pay attention
---
*GitHub: https://github.com/hydra-db/hydradb*