Smart Routing

Smart Routing

Smart Routing is an opt-in cross-provider fallback mechanism. When a request to your primary provider fails, the gateway can retry the same request against another provider in the same capability tier — without you rewriting the client code.

It is off by default. Every API key starts with Smart Routing disabled. You enable it per key from the dashboard, and you can turn it off at any time.

What Smart Routing does, and when it falls back

When Smart Routing is enabled for a given API key, the gateway attempts the request against the provider your model parameter maps to (the "original" provider). If that attempt fails with one of the following conditions, the gateway retries the request against the next provider in the fallback chain:

Any other 4xx error (such as 400, 401, 403, 404, 422) is treated as a client-side issue and is passed through to you unchanged — Smart Routing does not retry those.

Which providers participate

Smart Routing currently maps across Anthropic ⇄ OpenAI ⇄ DeepSeek, restricted to the same capability tier (high / mid / low). A fallback never downgrades to a lower tier.

Models from other providers — Gemini, GLM, Kimi, and others — are not in the fallback matrix. If the request to such a provider fails, the error is returned to you directly. There is no cross-provider retry for those.

If no fallback target is available (for example, the original provider is outside the matrix and failed), the gateway returns `502` with X-Served-By: none.

How to enable or disable

Smart Routing is controlled per API key from the dashboard:

1. Open Dashboard → your workspace → API Keys (/dashboard/{workspace-slug}/api-keys).

2. Find the key you want to configure.

3. Click the routing toggle in that key's row. The control is labeled "Cross-provider routing" (跨厂商路由 in Chinese).

4. The toggle takes effect immediately. When it shows "Routing enabled" (路由已开启), requests using that key participate in Smart Routing. When it shows "Routing disabled" (路由已关闭), requests go directly to the provider your model parameter specifies.

You can turn Smart Routing off at any time. Disabling it restores direct routing — no configuration change to your client is needed, and no request is retried to another provider while it is off.

The setting is stored on the API key (api_keys.routing_enabled, default 0). Requests using a key with the flag off never enter the fallback path.

How to identify which model actually served a response

When Smart Routing is active for the request, the response carries four X-Served-By* headers. They tell you exactly which provider and model produced the response body you received.

Example

curl -sD - -o /dev/null https://api.codegateway.dev/v1/chat/completions \
  -H "Authorization: Bearer $CODEGATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-6",
    "messages": [{"role": "user", "content": "hello"}]
  }'

Response headers (fallback occurred because the original provider returned a 5xx):

X-Served-By: openai:gpt-4o
X-Served-By-Fallback-Chain: anthropic,openai
X-Served-By-Original: anthropic:claude-sonnet-4-6
X-Served-By-Reason: 5xx

When the original provider succeeds, X-Served-By matches X-Served-By-Original and X-Served-By-Reason is none.

Streaming responses

For streaming (SSE) responses, these headers are sent before the body. You can read them first and decide — for example, log the actual model or abort — before consuming the stream.

curl -sN -D - https://api.codegateway.dev/v1/chat/completions \
  -H "Authorization: Bearer $CODEGATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-6",
    "stream": true,
    "messages": [{"role": "user", "content": "hello"}]
  }'

The X-Served-By* headers appear in the initial HTTP response headers, before any data: lines of the SSE stream.

Note: requests for Anthropic models via /v1/chat/completions use a cross-endpoint translation path that bypasses Smart Routing and returns no X-Served-By* headers. Use /v1/messages when you need to read them.

Impact on tool calls and output format

When a fallback occurs, the request is served by a different provider. Cross-provider responses are not guaranteed to behave identically. If you depend on any of the following, test your integration against the fallback path before relying on it in production:

• Tool call / function-call syntax — providers use different formats for tool definitions and tool-call responses.

• Stop reasons — the field that indicates why generation ended may use different values or naming across providers.

• Token counting — input/output token accounting can differ between providers for equivalent requests.

• Latency — a different provider may respond faster or slower than the original.

If `X-Served-By` differs from `X-Served-By-Original`, the response was produced by a different provider than the one you asked for. Automated pipelines that parse response structure (tool calls, stop reasons, token usage) should re-validate behavior whenever the served provider changes — or pin the path by leaving Smart Routing disabled.

Billing

Requests are billed at the rates of the provider that actually served the request — which is the provider shown in X-Served-By. If a fallback moved the request from one provider to another, you are charged the target provider's rate, not the original.

To audit what model served a given request, open Dashboard → your workspace → Logs (/dashboard/{workspace-slug}/logs). Each log entry shows the actual provider and model (actual_provider / actual_model) alongside the model you originally requested, so you can reconcile cost with usage after a fallback.