mesh-llm: Notes on Stitching the World Into One Giant GPU

> Source-level research note on C:\GitHub\mesh-llm\ (v0.72.1, 57 Rust crates + multi-language SDKs), dated July 21, 2026, written in a Feynman-style, de-AI-ified voice by the original author.

The Opening Question

The author has only an RTX 4060 (8 GB VRAM) and wants to run Qwen3-235B (~470 GB at F16). Normally impossible. mesh-llm's answer: link an idle MacBook, a Mac Studio, a used P40, even a Steam Deck into a mesh, slice the model by layers across nodes, and chain them into a machine with effectively 470 GB of memory. After reading the source, the author concludes: it doesn't cheat — but its real scope and fragile points must be stated honestly.

One-Sentence Positioning

mesh-llm = a decentralized LLM inference node binary + a control-plane protocol. Idle GPUs are discovered via Nostr announcements / mDNS LAN, connected with iroh (QUIC-based P2P tunnels), and exposed through an OpenAI-compatible http://localhost:9337/v1.

• vs llama.cpp: single-machine execution; mesh-llm does cross-machine collaborative inference.
• vs vLLM / TGI: single-node accelerators; mesh-llm is a cross-machine inference orchestrator.
• vs Petals: Petals is free-join swarm serving with Hivemind DHT; mesh-llm is operator-controlled with explicit topology planning and signed join tokens (SignedNodeOwnership, default 168 h validity).
• vs Exo: Exo mixes tensor parallelism (all-reduce) with Thunderbolt 5 RDMA (Apple Silicon only); mesh-llm is pure layer-pipeline, no tensor parallelism, covering CUDA/ROCm/Vulkan/Metal/CPU.

Control Plane: the Mesh Layer — "Manages People, Not Work"

The Mesh crates (identity, protocol, routing, hardware-profile, guardrails, system) do not execute inference. They manage: who a node is, how it speaks, what hardware it has, and request routing. A node's lifecycle (per docs/MESHES.md:42-75, mesh-llm-identity/src/ownership.rs:8-86):

1. Generate an owner keypair — owner_id = sha256(ed25519_verify_key). 2. Generate a 32-byte node key at ~/.mesh-llm/key (0o600). 3. Owner signs a NodeOwnershipClaim (cert + 7-day expiry); without it, no mesh membership. 4. Create (--model X --publish) or join (--join <signed-bootstrap-token>) via Nostr or mDNS. mDNS is LAN-only — public meshes need a Nostr relay. 5. Heartbeats every 60 s with gossip to 5 random peers; repeated failures broadcast STREAM_PEER_DOWN.

Routing: session-affinity first, round-robin fallback

select_model_target_from_candidates() (mesh-client/src/network/affinity.rs:458-523) computes a session_hash, a prefix_hash (tools + system messages), and a sticky hash, then resolves in order: session stickiness → prefix-hash LRU cache (TTL 20 min, 4096 entries) → forced prefix mode → sticky → round-robin. Rationale: hitting the same node reuses its prefix KV cache and skips a prefill, with unhealthy-node cooldown (30 s–5 min). Note: mesh-llm-guardrails is not failure recovery — it is an output/tool-call gatekeeper (tool-call retry caps, context compression at 90%).

Heterogeneous clusters: partially

Hardware is normalized to five backend flavors (CPU/CUDA/ROCm/Vulkan/Metal), but binaries ship per flavor — a Mac won't run a ROCm stage. Heterogeneity is usable for dispatching models by node capability, not GPU-level operator fusion. The official Qwen3-235B demo (Mac Studio + MacBook over iroh) excludes NVIDIA entirely.

Execution Plane: Skippy — Slice the Model Into N Segments

Skippy is plain layer-pipeline: a 40-layer model becomes contiguous segments on nodes A → B → C. Why so primitive? Robustness: tensor parallelism wants RDMA (rare), head parallelism is topology-sensitive; layer pipeline sends controlled-size hidden-state slices and tolerates node loss by re-election.

• Four planners (skippy-topology): even split; VRAM-weighted; package-aware scoring (cache affinity + missing-bytes + RTT penalty + availability); explicit user splits.
• Family capability table (STAGE_RUNTIME_LLAMA_FAMILY_EXPECTATIONS, lib.rs:298-813) covering 100+ llama.cpp architectures, flagging hybrid/recurrent models (jamba, lfm2, rwkv7, qwen3next) and recording forbidden split points (e.g., SharedKvProducerConsumer layers). This per-architecture table is the author's pick for the most underrated design.
• Inter-node transfer: hidden-state activations (ActivationDescriptor: dtype F32/F16/Bf16, layout, layer range, token counts, SHA-256). Roughly 8 KB/token for Llama-3-8B F16; ~512 MB per hop for Qwen3-235B 4-stage 32k prefill. More stages ⇒ longer prefill latency; decode is not amplified. There's an RTT penalty but no hard cap — 3 stages is sweet, 4+ becomes network-bound.
• Coordination: skippy-coordinator handles term-based election with ClaimFence and quorum n/2 + 1; actual topology execution lives in mesh-llm-host-runtime ("coordinator is the referee, mesh is the coach").
• Three layers of correctness: dev-time skippy-correctness baseline-vs-split token comparison; runtime BLAKE3 namespaced prefix_hash_with_namespace (model/topology/stage/layer/ABI/ctx/tokens) preventing cross-stage KV reuse; request validation rejecting wrong generations and path escapes (..).
• Known fragility (SKIPPY.md:641-645): active generations are not resumed across topology failure — computation already pushed downstream is silently dropped when a stale leader's in-flight LoadStage is fenced off. Unlike Exo's DiskEventLog replay, Skippy has no recovery layer; the docs don't hide this.

Model & Runtime Layer

Model lifecycle: model-ref parsing → resolver (local path → local dir → curated catalog → HuggingFace fallback) → artifact selection → download/cache with Range resume → packaging via HF Jobs REST API, which slices large models into {dist}-layers repos in the cloud — users never write their own layer-splitting scripts.

The three "runtime" crates are misleadingly named: host-runtime is the full node process (the mesh-llm CLI entry); native-runtime is a dlopen loader that picks which libllama.so flavor to load (CUDA rank 650 > Metal > ROCm > Vulkan > CPU); embedded-runtime is a 4-line re-export facade for SDK embedding. Backend switching requires a node restart.

The llama.cpp relationship is a fork + ABI rewrite: third_party/llama.cpp/ is patched to expose skippy.h, and skippy-ffi declares C symbols like llama_log_set, ggml_log_set, llama_model_quantize, skippy_model_open.

API & Console

• OpenAI-compatible endpoint on port 9337 (/v1/models, /v1/chat/completions, /v1/completions, /v1/responses, health checks).
• React SPA console on port 3131; plus MoA mixture-of-agents (a virtual mesh: model with three arbitration tiers), plugin hooks (on_mesh_event), and a panic-safe TUI.

Three Sharp Feynman Points

1. Solved: cross-machine layer-pipeline inference over heterogeneous consumer hardware with honest control-plane/execution-plane separation. Not solved: tensor parallelism, event-log recovery across topology failures, GPU-level fusion across vendors. 2. Genuinely clever: the per-architecture capability table with forbidden split points; prefix-affinity routing for KV reuse; cloud-side slicing via HF Jobs. Borderline cargo-cult: the guardrails naming (implies failover, isn't), and "three runtimes" that are really one process, one loader, and one facade. 3. Unknown: real-world multi-hop throughput under consumer ISP conditions, long-term KV-cache hit rates in production routing, and behavior beyond the documented 3–4 stage sweet spot — questions the source alone cannot answer.

Closing

mesh-llm is honest engineering: a deliberately conservative layer-pipeline design, source-verifiable claims, and documentation that admits its own failure modes. It delivers exactly what it promises — nothing more, and importantly, nothing less.

*Translated and edited from the original Chinese forum post on zhichai.net.*