Route models with the SDK gateway

The gateway is an SDK-local routing layer. It selects among provider adapters for one request; it is not a hosted proxy, an authentication boundary, or a billing service.

Use it when the application needs an explicit primary model, ordered fallbacks, normalized results, and inspectable attempt metadata. Keep calling a provider directly when one deployment and one model are already the desired contract.

Availability and stability

Capability	TypeScript	Python
Package or import	@zhivex-ai/gateway	Public zhivex_ai imports
Gateway contract	Stable	Stable
Text and object routing	Generate and stream	Generate and stream
Agent routing	runAgent() and streamAgent()	Not exposed by the Python gateway
Route-scoring metadata	Beta ergonomics	Included in the stable gateway result

Both gateways route the SDK’s portable model operations. They do not make a provider-native hosted tool, media endpoint, or realtime session portable. Check provider capabilities before adding a target.

TypeScript

Install the router separately; it is intentionally not re-exported from @zhivex-ai/sdk:

bun add @zhivex-ai/gateway @zhivex-ai/openai @zhivex-ai/anthropic

import { createGateway } from "@zhivex-ai/gateway";
import { createAnthropic } from "@zhivex-ai/anthropic";
import { createOpenAI } from "@zhivex-ai/openai";

const gateway = createGateway({
  adapters: {
    openai: createOpenAI({ apiKey: process.env.OPENAI_API_KEY }),
    anthropic: createAnthropic({ apiKey: process.env.ANTHROPIC_API_KEY })
  },
  maxFallbacks: 3,
  maxRetries: 1,
  maxTotalAttempts: 6,
  attemptTimeoutMs: 15_000,
  unknownCostPolicy: "reject"
});
const controller = new AbortController();

const result = await gateway.generate({
  primary: { provider: "openai", modelId: "gpt-4o-mini" },
  fallbacks: [{ provider: "anthropic", modelId: "claude-3-5-sonnet" }],
  messages: [{ role: "user", content: "Summarize this incident." }],
  requiredCapabilities: { tools: true },
  routingMode: "balanced",
  abortSignal: controller.signal
});

console.log(result.text, result.providerUsed, result.attempts);

The TypeScript gateway can also route structured output and agents. A streaming target can fall back only before provider output is exposed, so one client stream never mixes two provider transcripts.

Python

from zhivex_ai import (
    GatewayConfig,
    GatewayMessage,
    GatewayModelTarget,
    create_anthropic,
    create_gateway,
    create_openai,
)

gateway = create_gateway(
    GatewayConfig(
        adapters={
            "openai": create_openai(),
            "anthropic": create_anthropic(),
        },
        max_retries=1,
        attempt_timeout_ms=15_000,
        fail_on_missing_adapter=True,
    )
)

result = await gateway.generate(
    messages=[GatewayMessage(role="user", content="Summarize this incident.")],
    primary=GatewayModelTarget(provider="openai", model_id="gpt-5.6-terra"),
    fallbacks=[
        GatewayModelTarget(provider="anthropic", model_id="claude-sonnet-5")
    ],
    routing_mode="balanced",
)

print(result.text, result.provider_used, result.attempts)

Python additionally supports generate_object(), stream_text(), and stream_object(). It does not currently expose the TypeScript gateway’s agent-routing or total-attempt ceiling, so validate and cap externally supplied fallback lists in application code.

Design the route as policy

• Keep the primary, fallbacks, model ids, capability requirements, cost ceilings, retries, and timeouts in trusted server configuration.
• Require the original request shape. A vision request must route to a vision-capable target; never remove an attachment just to make a fallback compatible.
• Log every attempted or skipped target, plus request id, selected model, latency, normalized usage, and retryability. Redact prompts and provider error bodies before export.
• Decide explicitly whether a generated refusal is a final response. Python returns refusals by default and offers fallback_on_refusal=True; do not treat safety refusals as transient infrastructure errors accidentally.
• Forward cancellation and use downstream idempotency for agent tools or other side effects. A model fallback must not duplicate a payment, email, deployment, or write.

Application-owned boundary

The application still owns authentication, tenant isolation, provider credentials, rate limiting, spend enforcement, model allowlists, audit retention, and user-visible error mapping. Never accept an arbitrary provider, model id, retry count, or fallback chain from an untrusted client.

Continue with production architecture and errors and troubleshooting.