Route models with the SDK gateway

The gateway is an SDK-local routing layer. It selects among provider adapters for one request; it is not a hosted proxy, an authentication boundary, or a billing service.

Use it when the application needs an explicit primary model, ordered fallbacks, normalized results, and inspectable attempt metadata. Keep calling a provider directly when one deployment and one model are already the desired contract.

Availability and stability

Capability	TypeScript	Python
Package or import	@zhivex-ai/gateway	Public zhivex_ai imports
Gateway contract	Stable	Stable
Text and object routing	Generate and stream	Generate and stream
Agent routing	runAgent() and streamAgent()	Not exposed by the Python gateway
Route-scoring metadata	Beta ergonomics	Included in the stable gateway result

Both gateways route the SDK’s portable model operations. They do not make a provider-native hosted tool, media endpoint, or realtime session portable. Check provider capabilities before adding a target.

TypeScript

Install the router separately; it is intentionally not re-exported from @zhivex-ai/sdk:

bun add @zhivex-ai/gateway @zhivex-ai/openai @zhivex-ai/anthropic

import { createGateway } from "@zhivex-ai/gateway";
import { createAnthropic } from "@zhivex-ai/anthropic";
import { createOpenAI } from "@zhivex-ai/openai";

const gateway = createGateway({
  adapters: {
    openai: createOpenAI({ apiKey: process.env.OPENAI_API_KEY }),
    anthropic: createAnthropic({ apiKey: process.env.ANTHROPIC_API_KEY })
  },
  maxFallbacks: 3,
  maxRetries: 1,
  maxTotalAttempts: 6,
  attemptTimeoutMs: 15_000,
  unknownCostPolicy: "reject"
});
const controller = new AbortController();

const result = await gateway.generate({
  primary: { provider: "openai", modelId: "gpt-6-astra" },
  fallbacks: [{ provider: "anthropic", modelId: "claude-sonnet-5" }],
  messages: [{ role: "user", content: "Summarize this incident." }],
  requiredCapabilities: { tools: true },
  routingMode: "balanced",
  abortSignal: controller.signal
});

console.log(result.text, result.providerUsed, result.attempts);

The TypeScript gateway can also route structured output and agents. A streaming target can fall back only before provider output is exposed, so one client stream never mixes two provider transcripts.

Python

from zhivex_ai import (
    GatewayConfig,
    GatewayMessage,
    GatewayModelTarget,
    create_anthropic,
    create_gateway,
    create_openai,
    default_model_catalog,
)

gateway = create_gateway(
    GatewayConfig(
        adapters={
            "openai": create_openai(),
            "anthropic": create_anthropic(),
        },
        max_retries=1,
        attempt_timeout_ms=15_000,
        fail_on_missing_adapter=True,
        model_catalog=default_model_catalog,
    )
)

result = await gateway.generate(
    messages=[GatewayMessage(role="user", content="Summarize this incident.")],
    primary=GatewayModelTarget(provider="openai", model_id="gpt-6-astra"),
    fallbacks=[
        GatewayModelTarget(provider="anthropic", model_id="claude-sonnet-5")
    ],
    routing_mode="balanced",
)

print(result.text, result.provider_used, result.attempts)

Python additionally supports generate_object(), stream_text(), and stream_object(). It does not currently expose the TypeScript gateway’s agent-routing or total-attempt ceiling, so validate and cap externally supplied fallback lists in application code.

Catalog and cost evidence in Python

Python 0.22 uses an injected ModelCatalog to rank fallback targets from maintained recommendations instead of substrings such as pro, flash, or lite. The primary target always remains first. A cataloged target with missing or false required-capability metadata fails closed before adapter invocation; uncataloged targets retain the legacy compatibility path.

Inspect result.route_decision.target_evidence to see each requested and canonical model id, scoring source, recommendations, capabilities, lifecycle, resolved rate, and cost source. This explains the local policy decision; it is not provider certification or a final invoice.

When applying max_cost_per_1k_tokens, configure reviewed provider-and-model rates or typed catalog pricing:

gateway = create_gateway(
    GatewayConfig(
        adapters={"openai": create_openai(), "anthropic": create_anthropic()},
        model_catalog=default_model_catalog,
        model_costs_per_1k_tokens={
            "openai": {"gpt-5.6-luna": 0.25},
            "anthropic": {"claude-sonnet-5": 3.0},
        },
    )
)

The rates above are illustrative application configuration, not current price claims. Unknown, invalid, or over-ceiling prices are skipped before a provider call when a ceiling is set. provider_costs_per_1k_tokens remains accepted for compatibility but is deprecated because one provider can expose models with materially different prices.

Terminal attempt telemetry

Python 0.22 emits exactly one on_attempt payload after every executed retry or skipped target. It no longer emits the former success-shaped pre-call event. Each terminal payload includes attemptId, phase="finished", terminal=True, provider/model identity, target and retry indexes, measured latency, retryability, a machine-readable policy reason, a typed errorType, and a sanitized message.

Combine attemptId with the application request id because it is deterministic only inside one route. Consumers migrating from earlier versions must count terminal events and emit their own application-level start event if needed; never parse human-readable error text as routing state.

Design the route as policy

• Keep the primary, fallbacks, model ids, capability requirements, cost ceilings, retries, and timeouts in trusted server configuration.
• Require the original request shape. A vision request must route to a vision-capable target; never remove an attachment just to make a fallback compatible.
• Log every attempted or skipped target, plus request id, selected model, latency, normalized usage, and retryability. Redact prompts and provider error bodies before export.
• Decide explicitly whether a generated refusal is a final response. Python returns refusals by default and offers fallback_on_refusal=True; do not treat safety refusals as transient infrastructure errors accidentally.
• Forward cancellation and use downstream idempotency for agent tools or other side effects. A model fallback must not duplicate a payment, email, deployment, or write.

Application-owned boundary

The application still owns authentication, tenant isolation, provider credentials, rate limiting, spend enforcement, model allowlists, audit retention, and user-visible error mapping. Never accept an arbitrary provider, model id, retry count, or fallback chain from an untrusted client.

Continue with production architecture and errors and troubleshooting.

Adaptive routing and cost accounting

Tool-history-aware fallback can continue compatible OpenAI, Qwen, and DeepSeek tool loops without replaying resolved tools. Direct SDK calls retain raw tool-result serialization; Gateway can use explicit success/error envelopes. Portable DeepSeek/Qwen history replay defaults to no thinking and rejects explicit private-thinking replay.

In 1.16, auto structured-output mode resolves per destination, including native-to-prompted fallback, without restarting the tool loop. Stream errors are terminal before buffered tool execution; cancellation, cleanup, and Retry-After waits are bounded.

Version 1.20 adds opt-in local circuit breaking, explainable adaptive routing, and costAccounting request quotes and per-attempt reported valuations. Catalog pricing includes cache and long-context breakdowns, reasoning semantics, provenance, and unknown costs. Accounting does not change the legacy rate budget, and an accounting error must not retry a successful provider call. Reported token usage is preserved; only missing base counters are estimated.

These additions are TypeScript-specific. See release notes before adopting them in an existing Gateway.