Observe and evaluate agents

Observability explains what happened in a run. Evaluations check whether a reviewed behavior still holds. Use both: traces without tests only describe failures, while tests without production signals miss real operating conditions.

Stability by language

Language	Stable	Beta
TypeScript	Trace artifacts, trace summaries, cost estimates, agent replay, OpenTelemetry adapters, run ledgers, golden traces, and agent/workflow evaluation reports and gates	Agent telemetry event details and observer patterns outside the versioned OpenTelemetry adapter contract
Python	Agent replay is stable	Trace artifacts and collectors, OpenTelemetry integration, evaluation fixtures, repeated experiments, metrics, JSON/JUnit artifacts, and CI gates

For Beta surfaces, pin the SDK version, use public imports, cover the integration with tests, and review release notes before upgrading. Stable observability contracts still require application-owned privacy, exporter, sampling, retention, and access-control policy.

What to record

Capture operational facts that help correlate a failure without copying sensitive content:

• application request id, authenticated tenant reference, run id, and session id;
• agent id, provider, model id, status, and timestamps;
• latency, steps, tool-call count, and normalized error class;
• input, output, and total token usage when the provider reports them;
• approval counts and tool names without their arguments;
• workflow id, checkpoint sequence, transition, and idempotency key;
• retrieval source ids and index version when using RAG.

Prompts, model output, tool inputs and outputs, approval arguments, provider error bodies, credentials, and personal data should be excluded by default. Export content only to a reviewed destination after redaction, consent, tenant isolation, access control, and retention have been defined.

TypeScript: export a safe trace

After an agent run, create artifacts from its saved state without calling the model or tools again:

import {
  createAgentTraceArtifact,
  estimateAgentRunCost,
  summarizeAgentTrace
} from "@zhivex-ai/sdk";

const trace = createAgentTraceArtifact(result.state, {
  includeMessages: false,
  includeToolInputs: false,
  includeToolOutputs: false,
  includeApprovalArguments: false,
  includeOutputText: false,
  redaction: { includeEmails: true }
});

const summary = summarizeAgentTrace(trace, {
  latencyPercentiles: [0.5, 0.95]
});

const cost = estimateAgentRunCost(result.state, {
  inputCostPer1kTokens: 0.01,
  outputCostPer1kTokens: 0.03,
  currency: "USD"
});

console.log({ summary, cost });

The application owns the destination: structured logs, a queue, warehouse, dashboard, or SIEM. Zhivex does not ship a hosted observability UI.

Use createOtelAgentObserver() when OpenTelemetry lifecycle spans are appropriate. In TypeScript 1.6 the OpenTelemetry adapters are Stable and use a versioned, privacy-first GenAI mapping for model, agent, tool, and workflow telemetry. Configure the OpenTelemetry SDK lifecycle, exporters, resource attributes, sampling, storage, and content policy in your application; Stable mapping does not make a destination safe for prompts or tool payloads.

TypeScript: deterministic evaluation

Use hard expectations for safety and workflow invariants:

import {
  createAgentEvaluationFixture,
  createAgentEvaluationReport,
  runAgentEvaluationFixture
} from "@zhivex-ai/sdk";

const fixture = createAgentEvaluationFixture({
  name: "support-agent",
  dataset: [
    {
      name: "lookup-before-answer",
      input: { prompt: "Check ticket_123." },
      expectations: {
        status: "completed",
        outputContains: "ticket_123",
        toolCalls: ["lookupTicket"]
      }
    }
  ]
});

const evaluation = await runAgentEvaluationFixture(fixture, { agent });
const report = createAgentEvaluationReport(evaluation);

if (!evaluation.ok) process.exitCode = 1;
console.log(report);

Golden traces, run-ledger comparisons, and workflow evaluation gates are Stable TypeScript regression tools. Promote only intentionally reviewed runs—never arbitrary production traffic—and keep any model judge bounded, versioned, and separate from deterministic safety assertions.

Python: bounded experiments and CI gates

Python evaluations are beta and support repeated trials, bounded concurrency, baseline comparison, custom metrics, and strict JSON/JUnit artifacts:

from zhivex_ai import (
    AgentEvaluationCase,
    AgentEvaluationExpectations,
    AgentEvaluationGate,
    run_agent_evaluation_experiment,
)

dataset = [
    AgentEvaluationCase(
        name="refund-policy",
        prompt="Can this order be refunded?",
        expectations=AgentEvaluationExpectations(output_contains="review"),
    )
]

experiment = await run_agent_evaluation_experiment(
    variants={"baseline": baseline_agent, "candidate": candidate_agent},
    baseline="baseline",
    dataset=dataset,
    gates=[
        AgentEvaluationGate(
            "pass_rate",
            minimum=0.95,
            max_regression=0.01,
        )
    ],
    repetitions=5,
    max_concurrency=4,
)

print(experiment.to_json())
raise SystemExit(0 if experiment.ok else 1)

Prefer an agent factory when trials run concurrently. Reusing one agent instance is safe only when its model, memory, tools, and injected dependencies are reentrant.

The Python CLI can write machine-readable artifacts for a CI system:

zhivex eval my_app.agents:support_agent \
  --dataset evals/support.json \
  --repetitions 5 \
  --max-concurrency 4 \
  --min-pass-rate 0.95 \
  --output-json artifacts/evaluation.json \
  --output-junit artifacts/evaluation.xml

Importing the supplied module executes application code. Run the CLI only against trusted modules in an isolated CI environment.

Build an evidence ladder

Keep these checks separate because each proves something different:

1. Unit contract: deterministic model and tool fixtures prove local orchestration.
2. Replay: saved state can be inspected without another provider call.
3. Evaluation: datasets and expectations catch behavioral regressions.
4. Authenticated smoke: the exact provider, model, region, credentials, and operation work now.
5. Release evidence: the tested code is the exact commit and package artifact being promoted.
6. Production monitoring: real latency, errors, cost, safety, and quality remain inside policy after deployment.

An offline passing evaluation is not live-provider certification. A successful provider smoke is not a general quality evaluation.

Dataset and privacy policy

• Keep datasets, judge prompts, expected tool use, and thresholds in code review.
• Redact production data before converting it into fixtures or golden traces.
• Record provider and model configuration when comparing variants.
• Never use one model judge as the only approval for regulated or financially material behavior.
• Keep exact expectations for authorization, approval, tool order, schemas, and terminal status even when adding a probabilistic judge.
• Apply access control, encryption, retention, deletion, and legal basis to trace and evaluation artifacts.

See Production architecture for the runtime boundary and Workflows for persisted workflow state, replay, and workflow-specific evaluation.

Full references

• TypeScript observability guide
• TypeScript observability example
• Python observability guide
• Python evaluation guide