Google API Gateway Adds Model Routing: A Managed LiteLLM Replacement Locked to Model Garden

Forum topic · 小凯 · 2026-08-05

Summary

Google API Gateway has introduced model routing in preview (released August 3, 2026), positioned as a managed alternative to client-side LLM proxies such as LiteLLM. Clients send OpenAI-compatible POST /chat/completions requests to a serverless gateway URL; the gateway reads the model field in the JSON body, matches it against routing rules declared via two OpenAPI 3.x extensions, and on-the-fly transcodes the payload into Vertex AI prediction schema before dispatching to Model Garden endpoints. Unmatched requests fall back to a defaultModel. The feature only routes to models hosted in Vertex AI Model Garden, including Gemini, Claude, and open-weight gpt-oss MaaS models — it does not connect directly to Anthropic or OpenAI. Routing is static name-based with no cost, latency, or quality-based scheduling. Implications for AI coding harnesses: zero client-side changes, centralized auth, quotas, and observability at the edge, but it is a lightweight offering distinct from the heavier Apigee AI Gateway. Pricing follows standard API Gateway tiers; preview limitations include no error on missing model field.

Overview

Google API Gateway introduced model routing on August 3, 2026 (preview, v1). Google explicitly positions the feature as *"a managed alternative to client-side proxies such as LiteLLM,"* replacing self-hosted routing layers with a serverless gateway URL.

How It Works

The mechanism is straightforward:

1. Client sends an OpenAI-compatible request — POST /chat/completions — to the gateway. 2. Gateway reads the model field in the JSON body. 3. Gateway matches the value against routing rules declared in an OpenAPI 3.x spec. 4. The request is transcoded on-the-fly into the Vertex AI prediction schema and dispatched to a Model Garden endpoint. 5. Unmatched requests fall back to a configured defaultModel.

Routing configuration uses two OpenAPI 3.x extension fields. Example target models shown in the docs:

google/gemini-3.5-flash-lite
anthropic/claude-opus-4-7
openai/gpt-oss-120b-maas (open-weight gpt-oss via MaaS, not closed-source GPT)

Key points

Scope is narrower than reported. Chinese-language coverage framed this as "cross-vendor routing." It is not. All three example vendors are reachable only through Vertex AI Model Garden; the gateway does not connect directly to api.anthropic.com or api.openai.com.
Scheduling is static. Preview behavior: *"routes based exclusively on the model tag or name in the payload."* No cost-, latency-, or quality-based intelligent dispatch.
Single hostname constraint. All models inside one router must share the same host. Private/self-hosted models are not supported.
Operational value for coding harnesses. Enterprises that previously had to run containers, autoscale, and manage keys to feed models into agents can now point OpenAI SDKs at an edge gateway URL. Auth, quotas, and usage telemetry are centralized.
Not an Apigee replacement. True cross-vendor direct connections, semantic caching, and token budgets remain on the heavier Apigee AI Gateway product line. This is a lightweight offering and is locked to Google-hosted models.
Streaming & protocol limits. SSE response streaming is supported; request-side streaming, gRPC, WebSockets, and Gemini Live are not. Request timeout ceiling is 3600 seconds. VPC Service Controls is not supported.
Documented preview defect. If the request body omits the model field, the gateway does not return an error — it falls through to error handling.
Low-key launch. No standalone official blog post; the feature is documented only via Release Notes and product docs.

Pricing

Standard API Gateway tiers apply to routing calls; model tokens are billed separately via Vertex AI:

| Tier | Price | |---|---| | 0 – 2M calls/month | Free | | 2M – 1B calls | $3.00 / million | | > 1B calls | $1.50 / million |

Reported "Chained" Architecture

Chinese reports described a chain where requests exit via Agent Gateway and then pass through API Gateway for model dispatch. This two-step topology has no corresponding description in either official doc; treat it as speculation.

Sources

API Gateway Release Notes (2026-08-03): https://cloud.google.com/api-gateway/docs/release-notes
Model routing overview: https://docs.cloud.google.com/api-gateway/docs/model-routing-overview
Configure model routing: https://docs.google.com/api-gateway/docs/model-routing-configure
Pricing: https://cloud.google.com/api-gateway/pricing