Summary
GLM (Graph-CoT with Multi-Agent and Efficient LLM Serving) is a framework that combines multi-agent collaboration with system-level LLM serving optimizations to improve large-scale graph reasoning. It addresses key limitations of single-agent Graph Chain-of-Thought systems, including the lost-in-the-middle problem, repeated context re-encoding, high token costs, long inference latency (up to 40 seconds), and low throughput. GLM decomposes reasoning into four specialized components: a Classification Agent, a Reasoning Agent, an Action Agent that generates executable Python code, and a Graph RAG Retriever, coordinated through a shared Notebook state mechanism. On the serving side, it introduces vertex-centric KV-cache management, a four-tier priority-based cache eviction policy, and pipelined execution that overlaps graph retrieval with LLM decoding, built on top of vLLM. On the GRBench benchmark spanning academia, e-commerce, literature, healthcare, and law, GLM achieves up to 38% higher accuracy than Graph-CoT and up to 62% higher accuracy than Text RAG, while reducing token costs by 95.7%, cutting inference latency by 90.3%, and improving throughput by up to 15.1x.
This post introduces GLM (Graph-CoT with Multi-Agent and Efficient LLM Serving), a multi-agent framework co-designed with an efficient LLM serving architecture for large-scale graph reasoning over knowledge graphs.
Key Points
Problems with Existing Single-Agent Graph-CoT Systems
- All reasoning functions (classification, retrieval, inference, action generation) are packed into one large prompt handled by a single LLM, causing:
- Lost-in-the-middle: long contexts make LLMs ignore key information.
- Repeated context re-encoding: entire context is re-processed every iteration, wasting computation.
- Accuracy vs. cost trade-off: broad retrieval pushes token costs above $3 per query on some commercial models; restricted retrieval drops accuracy below 50%.
- End-to-end latency up to 40 seconds and low throughput.
GLM's Multi-Agent Architecture
- C-Agent (Classification): routes queries — deterministic queries go through a fast path directly to the retriever; non-deterministic queries enter an iterative reasoning loop.
- R-Agent (Reasoning): analyzes the shared state and plans what information is needed next.
- A-Agent (Action): translates reasoning plans into executable Python code for the graph database.
- Graph RAG Retriever: executes the code and appends results back to shared state.
- A central "Notebook" enables selective, lightweight context sharing, avoiding context bloat. The iterate-plan → generate-code → retrieve → update loop runs until the answer is ready or an iteration cap is reached.
LLM Serving Co-Design (built on vLLM)
1. Graph-CoT-aware KV-cache management: caches a "vertex chunk" (a node plus its 1-hop neighbors), greatly improving cross-query cache reuse and skipping expensive prefill computation.
2. Priority-based eviction (4 tiers): system instructions (never evicted) → active-session notebooks → completed query notebooks → temporary intermediate outputs (evicted first), significantly raising hit rates over plain LRU.
3. Pipelined execution: graph retrieval requests fire asynchronously as soon as retrieval-call lines are decoded, overlapping I/O with LLM decoding to hide retrieval latency.
- Also includes a global LRU cache mapping text queries to node IDs, plus fault tolerance: code-execution errors are fed back to A-Agent for self-correction, and failed requests can recover from cached KV state.
Evaluation (GRBench benchmark: academia, e-commerce, literature, healthcare, law)
| Metric | Result vs. Graph-CoT |
|---|---|
| Answer accuracy | up to +38% |
| Accuracy vs. Text RAG | up to +62% |
| Token cost | -95.7% (from >40,000 to ~1,538–2,974 tokens/query) |
| Inference latency | -90.3% (from 11–39s to 2.8–5.9s) |
| Throughput | up to 15.1x (0.6–2.2 → 6.8–9.1 queries/sec) |
- GLM typically needs only 2–3 LLM calls per complex query, versus 9–14 for Graph-CoT.
Applications and Future Directions
- Use cases: academic knowledge-graph QA, e-commerce recommendation, healthcare and legal knowledge reasoning.
- Future work: generalization to other structured data (relational DBs, JSON), distributed deployment, dynamic graphs with incremental KV-cache updates, and richer agent interaction modes (debate, negotiation, reinforcement learning).
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/176360536