GLM: A Multi-Agent Framework with Efficient LLM Serving for Large-Scale Graph Reasoning

Key points

• Problem: Single-agent Graph-CoT systems suffer from the "lost-in-the-middle" effect, repeated context re-encoding, high token costs (over $3 per query on some commercial models), up to 40-second latency, and accuracy below 50% when retrieval is restricted to control costs.
• Approach: GLM splits reasoning into specialized agents and co-designs the LLM serving layer for graph workloads.

Architecture

| Agent | Role | Responsibility | | :--- | :--- | :--- | | C-Agent | Gatekeeper/router | Classifies queries as deterministic (fast path via direct retrieval) or non-deterministic (iterative reasoning loop) | | R-Agent | Planner | Analyzes Notebook state, formulates high-level reasoning plans | | A-Agent | Engineer | Translates plans into executable Python code for graph interaction | | Graph RAG Retriever | Executor | Runs generated code against the graph database, appends results to the Notebook |

• Notebook mechanism: a centralized, structured state store enabling selective, lightweight information sharing between agents instead of passing full conversation histories — reducing context length, token usage, and noise.
• Iterative loop for non-deterministic queries: plan (R-Agent) → code generation (A-Agent) → retrieval and Notebook update (retriever) → repeat until sufficient information is gathered or an iteration cap is reached.
• Built on vLLM with customizations for graph-aware caching and pipelined execution; includes a global LRU cache mapping text queries to node IDs, self-correcting code execution (errors fed back to A-Agent for retry), and request-level fault recovery from cached KV states.

LLM serving optimizations

1. Vertex-centric KV-cache management: caches "vertex chunks" — a node plus all its 1-hop neighbors — as reusable units, dramatically raising cross-query hit rates and skipping expensive prefill computation for hot graph regions. 2. Priority-based eviction (four levels):

• Priority I (highest): system instructions / agent role definitions — never evicted
• Priority II: active session Notebook content — demoted after session end
• Priority III: completed query Notebooks — evicted under memory pressure before I/II
• Priority IV: temporary intermediate outputs (reasoning steps, code fragments) — evicted first
• This replaces plain LRU and substantially improves cache hit rates.
3. Pipelined execution: once decoding reaches a retrieval call (e.g., RetrieveNode("...")), an async retrieval request fires while the LLM continues decoding non-dependent code, hiding graph-database I/O latency from the critical path.

Evaluation (GRBench benchmark)

GRBench covers five domains: academia, e-commerce, literature, healthcare, and law, with queries ranging from single-hop lookups to multi-hop aggregation and logical reasoning. Baselines: Graph-CoT (single-agent, iterative) and Text RAG (flat text retrieval).

| Metric | GLM | Graph-CoT baseline | Improvement | | :--- | :--- | :--- | :--- | | Answer accuracy | up to +38% | — | +38% vs Graph-CoT | | Accuracy vs Text RAG | up to +62% | — | +62% | | Token cost | 1,538–2,974 tokens/query | > 40,000 tokens/query | −95.7% | | Inference latency | 2.8–5.9 s | 11–39 s | −90.3% | | Throughput | 6.8–9.1 queries/sec | 0.6–2.2 queries/sec | up to +15.1x |

• LLM calls per complex query drop from 9–14 (Graph-CoT) to typically 2–3 (GLM).
• Code-based retrieval is more precise than natural-language instructions, and specialized short prompts avoid context dilution — both contribute to accuracy gains.

Applications

• Academic knowledge graph QA: multi-hop questions over papers, authors, venues, and citations.
• E-commerce and recommendations: co-purchase analysis, competitor product search, real-time personalized recommendations.
• Healthcare and law: reasoning over structured medical and legal knowledge bases where accuracy and explainability are critical.

Future directions

• Generalization and scaling: adapting the framework to other structured data (relational databases, JSON) and distributed multi-GPU deployments.
• Dynamic graphs and real-time reasoning: incremental KV-cache updates when nodes/edges change.
• Richer agent interaction: debate/negotiation mechanisms, self-evolving agents, and reinforcement learning for agent decision-making.

Comparison with related work

Unlike earlier Graph-CoT research focused on prompt-level algorithm design, GLM's key contribution is addressing efficiency and scalability bottlenecks at the system level through algorithm–infrastructure co-design. It also extends the broader multi-agent LLM trend (e.g., AutoGPT, ChatDev) to graph reasoning with highly specialized agent roles and demonstrates that serving-layer co-optimization is essential for unlocking multi-agent architecture potential.