Higress Review: Building an AI Switchboard for Your Company

By the second year of AI adoption, most companies look like this: the WeCom bot backend holds a Doubao key; Codex config points to some employee's personal OpenAI account; OpenCode connects to a Qwen instance on a test server; Feishu bots each run their own backend. Keys are scattered across seven or eight places, bills live in separate vendor consoles, and switching models means editing every app's code. When something goes wrong—say, a leaked key—nobody can say who was using it.

This is the multi-plug power strip problem. Alibaba's open-source Higress wants to be the breaker box.

Identity Check

• GitHub: higress-group/higress, 9,591 stars, written in Go, latest release v2.2.5 (Oct 4), active commits.
• Open-sourced October 2022, born from Alibaba's gateway team that handled major shopping-festival traffic.
• Technical base: Istio control plane + Envoy data plane; plugins run as WebAssembly (Go, Rust, JS supported) and can be updated without restarting the gateway.
• CNCF sandbox project. Production adopters include Ant Digital Technologies, Kuaishou, Trip.com, and Vipshop, listing "LLM Gateway, MCP Gateway" as use cases.

The core capability in one sentence: change routes without dropping connections. Nginx reloads cut long connections—fine for web pages, disastrous for streaming AI responses that run tens of seconds. Envoy hot updates plus Istio routing make live reconfiguration the default. This makes Higress naturally suited to sit at the AI traffic choke point.

Layer 1: Unified Model Access

Higress's ai-proxy plugin uses OpenAI's API format as the lingua franca. Counting the plugin source: 38 vendor adapters, including OpenAI, Claude, Gemini, Bedrock, Tongyi, Zhipu, DeepSeek, Doubao, Hunyuan, and Moonshot. Requests are auto-detected by protocol (/v1/chat/completions parsed as OpenAI, /v1/messages as Claude) and converted on the fly when the target vendor expects a different format. Model names support regex mapping, e.g., rewriting all gpt-prefixed requests to corresponding internal Qwen models.

Integration is config-only, no code changes:

• Codex: edit ~/.codex/config.toml, add a [model_providers] section with name, base_url, env_key pointing at the gateway—third-party endpoints are officially supported.
• OpenCode: edit the provider section in opencode.json to set baseURL; the docs state this option is for "proxy services or custom endpoints."

For WeCom bots and Feishu bots with self-built backends, only the backend's model call line changes. All apps then see one address and one format; model swaps happen at the gateway, and vendor failures are handled by token-pool health checks.

Layer 2: Token Is the New Bill

Old gateways rate-limit by request count—meaningless when one request is a 100-token Q&A and another a 50,000-token analysis. Higress does token-level quotas: each consumer (a re-issued API key) gets a token budget that stops when exhausted; QPS limiting is secondary. Also included: semantic caching (similar questions return cached answers, skipping inference) and content safety plugins (prompt-injection detection, sensitive content identification). Dashboards settle token usage per consumer—a per-app itemized AI bill.

Layer 3: Tools Go Through the Switchboard Too

This layer covers MCP—the protocol AIs use to call tools—and is where Higress differentiated itself in 2025 from OneAPI and LiteLLM. Three tiers:

1. The gateway directly hosts MCP servers: any REST API becomes an MCP service via console config, no code—templates declare each tool's request/response formats; databases are built-in. 2. Batch conversion via the open-source openapi-to-mcpserver tool using OpenAPI documents. 3. A public marketplace at mcp.higress.ai with ready-to-try MCP services.

For enterprises, the real value: tool auth, rate limiting, and call auditing concentrate at the gateway instead of scattering across agent configs. Agents hold only short-lived tokens; real credentials stay locked behind the gateway.

The Full Picture

| Dimension | Direct connections | Through Higress | |---|---|---| | Access | Each app's own keys/SDKs | Unified OpenAI-compatible endpoint + consumer keys | | Model swap | Edit every app | Gateway-side mapping, hot-updated | | Billing | Separate vendor consoles | Token details per consumer | | Failures | Each app's own retries | Token-pool health checks auto-failover | | Tools | Agents install MCP separately | Gateway-hosted unified MCP endpoint | | Credentials | Scattered in apps | Locked at gateway, calls logged |

App-layer clients—WeCom bots, Feishu bots, Codex, OpenCode—become extensions on the switchboard, backed by vendor models, self-hosted vLLM clusters, and MCP tools converted from existing services.

Author's Judgments

1. The barrier isn't "can forward"—OneAPI and friends already do that. The dividing line is the data plane: uninterrupted streaming, plugin hot updates, and a foundation built for hundreds of thousands of QPS. Using Envoy as the kernel reuses a decade of engineering, which is why Higress can stack MCP hosting, token quotas, and content safety on top. 2. The "AI entry point" is becoming infrastructure. Whoever controls the gateway controls model billing, tool permissions, and call records. The buyer is the IT department—the first time all AI traffic passes through a manageable, auditable, accountable place. This round's difference from the API gateway era: the metering unit changed from requests to tokens. 3. Caveats: only Ant Digital, Kuaishou, Trip.com, and Vipshop run LLM/MCP gateways in production per the ADOPTERS file; the "agent gateway" category is still under Testing (Autohome and others are trialing exactly the setup described here). Community edition stability docs say "handle it yourself"; fully managed requires Alibaba Cloud enterprise edition. Think hard about the line between open-source-free and production-accountability when choosing.

Minimal Validation Path

No K8s needed. One Docker command starts all-in-one (with console), configure one ai-proxy route to any model, point Codex's base_url at it, then register an internal REST API as an MCP service in the console. One afternoon: six extensions, two tools, one token ledger—the switchboard is live.

• GitHub: github.com/higress-group/higress (9,591 stars, Apache-2.0, CNCF sandbox)
• MCP marketplace: mcp.higress.ai