Observability, Cost, and Performance

Start with operational questions and objectives

Write the questions operators must answer: Is the right model serving? Are users receiving valid results? Which slices fail? Where is time spent? What is queued or rejected? Which resource is saturated? Did an update change quality, security, cost, or recovery? Can an incident be scoped and reproduced?

Define indicators and objectives for availability, successful outcomes, policy outcomes, latency distributions, queue time, throughput, rejection, error, recovery, and budget. A dashboard without a decision or owner becomes decorative telemetry.

Correlate logs, metrics, traces, events, and exact identities

Use a request or run identifier to connect gateway, adapter, inference server, runtime, tool, policy, and application events. Record service and instance identity, exact model and runtime bundle, route, operation, status, error category, policy disposition, input and output size measures, queue and processing time, resource observations, and release identifier.

OpenTelemetry semantic conventions provide shared names and stability metadata for telemetry concepts. Version the convention and application fields you adopt. Do not assume every generative-AI field is stable or available across runtimes; preserve an application-owned core and explicit adapters.

Minimize sensitive content and control telemetry access

Default to metadata and derived measurements rather than raw prompts, responses, retrieved documents, credentials, headers, or personal identifiers. If sampled content is necessary for an approved purpose, minimize, redact, encrypt, restrict access, record consent or authority, set short retention, audit use, and test deletion.

Threat-model telemetry pipelines, collectors, exporters, stores, dashboards, alerts, and support bundles. A secure serving path can still disclose data through verbose errors, traces, exception bodies, labels, exemplars, or high-cardinality identifiers.

Measure distributions, saturation, and attributable cost

Record latency and size as distributions with suitable histogram resolution or buckets, not only averages. Prometheus documents that precomputed summary quantiles are generally not aggregatable across replicas, while histograms support later aggregation subject to their representation. Preserve counts, failures, and the population behind every percentile.

Measure queue time, time to first output, processing and streaming time, throughput, cancellations, retries, cache behavior, accelerator and host utilization, memory headroom, storage and network activity, and energy or cloud charges where available. Attribute costs by bounded tenant, workload, route, model, and release without using unbounded labels.

Alert on symptoms with actionable context

Alert on user-visible objective burn, critical safety or security outcomes, sustained queueing, rejection, errors, memory pressure, unavailable replicas, model-identity mismatch, telemetry loss, cost anomalies, and failed backup or rollback checks. Attach owner, severity, runbook, evidence links, and safe first actions.

Test alert firing, routing, deduplication, suppression, escalation, and recovery. Preserve baseline and deployment annotations so an operator can compare before and after a change. Missing telemetry is a condition to detect, not evidence that the service is healthy.

Practice activity

Build a privacy-conscious observability contract

Write ten operator questions and map each to signals, exact identities, dimensions, owner, decision, and retention.
Define request, model, release, policy, latency, queue, error, saturation, and cost fields with stability and cardinality limits.
Classify every proposed field as required metadata, optional sensitive content, prohibited secret, or derived measurement.
Design latency and size histograms, objective calculations, and cost allocation for three replicas without averaging quantiles.
Create five alerts and paper-test firing, routing, evidence, runbook, escalation, and recovery.

What to produce

An operator-question-to-signal matrix and versioned telemetry field contract.
A privacy, access, retention, redaction, cardinality, sampling, and deletion review.
Objective, distribution, cost, dashboard, and alert fixtures with tested-paper incident and recovery links.

Reflect before continuing

Which telemetry field is easiest to collect but hardest to justify retaining?