Build multimodal and realtime features

Multimodal support is operation- and model-specific. A provider may accept images in text generation without exposing image generation, transcription, speech, video, or realtime through the same model.

Start from the shared contract when one exists, then isolate provider-native clients and model-specific realtime configuration behind an application-owned service.

Availability and stability

Surface	TypeScript	Python
Image/file input for generation	Shared content contract; model dependent	Stable foundation contract; model dependent
Text and multimodal embeddings	Stable shared helpers; adapter dependent	Stable embed* and embed_content* helpers
Transcription and speech	Stable shared helpers	Beta, provider dependent
Image, video, and music generation	Stable shared helpers; provider dependent	Google native media clients are beta; other media paths are provider-specific
Realtime/live sessions	Stable shared lifecycle; provider/model dependent	Experimental

This table describes SDK API stability, not live availability. Region, account approval, quota, preview status, file limits, output formats, and exact model support still need an authenticated smoke for the release you deploy.

Portable audio in TypeScript

import { readFile } from "node:fs/promises";
import { transcribeAudio } from "@zhivex-ai/sdk";
import { createOpenAI } from "@zhivex-ai/openai";

const openai = createOpenAI({ apiKey: process.env.OPENAI_API_KEY });
const audio = await readFile("meeting.wav");

const result = await transcribeAudio({
  model: openai.transcriptionModel!("gpt-4o-mini-transcribe"),
  audio: {
    data: new Uint8Array(audio),
    mediaType: "audio/wav",
    filename: "meeting.wav"
  },
  timeoutMs: 60_000
});

console.log(result.text);

The same shared layer exposes generateSpeech(), streamSpeech(), generateImage(), generateVideo(), and generateMusic(). Each helper checks the selected model capability and fails explicitly when the adapter does not implement it.

Multimodal input in Python

from pathlib import Path

from zhivex_ai import FilePart, ModelMessage, TextPart, create_gemini, generate_text

gemini = create_gemini()
audio_bytes = Path("call.mp3").read_bytes()
result = await generate_text(
    model=gemini("gemini-2.5-flash"),
    messages=[
        ModelMessage(
            role="user",
            parts=[
                FilePart(
                    data=audio_bytes,
                    media_type="audio/mpeg",
                    filename="call.mp3",
                ),
                TextPart(text="Summarize the decisions in five bullets."),
            ],
        )
    ],
)

For speech-to-text and text-to-speech, Python exposes transcribe_audio() and generate_speech() as beta helpers. For Gemini or Vertex image, video, and music jobs, use the beta native clients returned by provider.images(), provider.videos(), and provider.media().

Portable versus native media

Prefer portable helpers when the application cares about a normalized result and can tolerate capability-based provider selection. Use native clients when the feature requires provider lifecycle operations such as uploads, file ids, long-running video jobs, context caches, progressive image events, or provider-specific controls.

Keep the native code in one service and return an application-owned record:

media request
  -> authenticated application policy
  -> portable helper or provider-native client
  -> object storage
  -> application media id + metadata

Do not persist raw provider URLs as your only durable reference. They may be temporary, authenticated, region-specific, or subject to provider retention rules.

Open a TypeScript realtime session

The shared TypeScript session and live-agent lifecycle is Stable. Transport, authentication, event ordering, interruption, audio framing, tools, and model availability still differ from buffered generation and remain provider-scoped.

import { tool } from "@zhivex-ai/sdk";
import { createOpenAI } from "@zhivex-ai/openai";
import { z } from "zod";

const openai = createOpenAI({ apiKey: process.env.OPENAI_API_KEY });
const session = await openai.realtimeModel!("gpt-realtime").connect({
  instructions: "Keep answers short.",
  tools: {
    weather: tool({
      name: "weather",
      schema: z.object({ city: z.string() }),
      execute: async ({ city }) => ({ city, forecast: "sunny" })
    })
  }
});

await session.sendText("How is Madrid today?");

for await (const event of session.eventStream()) {
  if (event.type === "realtime-text-delta") {
    process.stdout.write(event.textDelta);
  }
}

The shared adapters currently cover OpenAI, Azure OpenAI, Gemini, Vertex, and Qwen. Gemini, Vertex, and Azure OpenAI can also accept image frames through sendMedia() on supported models. Qwen defaults to text output; set voice or outputAudioMediaType when audio output is required.

Compose a live agent

streamLiveAgent() adds local tools, approval policies, guardrails, telemetry, and optional state persistence over a realtime-capable model:

import { streamLiveAgent, tool } from "@zhivex-ai/sdk";
import { createGemini } from "@zhivex-ai/gemini";
import { z } from "zod";

const gemini = createGemini({ apiKey: process.env.GEMINI_API_KEY });
const live = streamLiveAgent(
  {
    id: "voice-weather",
    model: gemini.realtimeModel!("gemini-live-2.5-flash-native-audio"),
    instructions: "Speak briefly and use tools when needed.",
    tools: {
      weather: tool({
        name: "weather",
        schema: z.object({ city: z.string() }),
        execute: async ({ city }) => ({ city, forecast: "sunny" })
      })
    }
  },
  {
    prompt: "How is Madrid today?",
    realtime: {
      outputAudioMediaType: "audio/pcm",
      outputAudioTranscription: true
    }
  }
);

for await (const chunk of live.textStream) process.stdout.write(chunk);
const result = await live.collect();
console.log(result.outputText);

Realtime security boundary

• Mint browser-scoped ephemeral credentials on the server only where the provider supports them.
• Keep long-lived provider keys on the server. Some adapters intentionally support server-side realtime only.
• Use HTTPS/WSS trusted endpoints. Never derive allowUnsafeEndpoints or a credentialed endpoint from untrusted input.
• Bound incoming frame size, session duration, buffered audio, tool concurrency, and reconnect attempts. The browser-safe default transport rejects frames above 16 MiB unless explicitly configured.
• Treat partial transcripts and partial images as transient events unless the application deliberately persists them.
• Require application-owned identity, authorization, expiration, audit, and one-time consumption for side-effect approvals.
• Certify the exact provider, model, audio configuration, tool continuation, and connection closure before deployment. Stable SDK types do not prove live provider availability.

Data and safety boundary

• Verify actual bytes, MIME type, size, duration, dimensions, and decoding before sending media to a provider; filenames and client-declared content types are untrusted.
• Enforce tenant-scoped object keys, signed access, encryption, malware scanning where appropriate, retention, deletion, and region policy in application storage.
• Obtain the necessary consent for recording, transcription, voice generation, face or biometric processing, and training use. Avoid retaining media when derived text or metadata is enough.
• Strip location, camera, author, and other metadata unless required. Do not log base64 bodies, audio frames, signed URLs, or raw transcripts by default.
• Apply output moderation and provenance rules appropriate to the generated medium. Capability checks do not establish policy compliance.

See providers, production architecture, and errors and troubleshooting before deploying a multimodal route.