Project 42Start learning
Module 10 of 12 · 52 min

Observe Agent Quality, Cost, and Risk

Instrument agent runs with privacy-preserving traces, metrics, logs, and quality signals that support diagnosis without turning telemetry into a sensitive-data archive.

Core concept

By the end

You will be able to

  • Design traces that connect agent, model, retrieval, tool, approval, handoff, and verification spans.
  • Measure quality, safety, reliability, latency, token use, cost, and escalation by meaningful slices.
  • Apply data minimization, redaction, access, retention, sampling, and deletion controls to telemetry.
  • Use traces to localize failure and route human escalation without exposing hidden reasoning.
01

Use traces, metrics, logs, and evaluations together

A trace reconstructs one run and its parent-child operations. Metrics reveal trends and service objectives. Logs capture discrete operational events. Evaluations judge behavior against a rubric. None replaces the others.

Start from decisions the team must make: detect a regression, locate a failed boundary, control spend, explain an escalation, or prove a postcondition. Collect only signals that support those decisions.

02

Preserve causal structure across the run

Give the workflow a trace ID and each model call, retrieval, tool call, approval, guardrail, handoff, verifier, and recovery step a span with parent linkage and timing. Record stable workflow, agent, prompt, model, tool, policy, and content versions.

Use low-cardinality status and failure labels for aggregation, while keeping detailed evidence behind access-controlled references. Correlation IDs should connect provider requests and external side effects without storing secrets.

Portable span envelope
text
traceId, spanId, parentSpanId
operation: agent | model | retrieval | tool | approval | handoff | verify
versionRefs: workflow, prompt, model, tool, policy, content
start, end, status, failureClass
usage: tokens, estimatedCost, retries
resultRef: redacted evidence or secure pointer
03

Measure outcomes, not activity alone

Track task success and verified completion alongside policy violations, unsafe attempts, user corrections, escalations, abandonment, tool and retrieval failure, retry rate, latency percentiles, token use, and cost per successful task.

Segment by workflow version, model, tool, user journey, risk class, language, and other supported groups. Alert on error-budget burn, critical policy events, and material quality change rather than raw request volume alone.

04

Assume model and tool content is sensitive

Inputs, outputs, tool arguments, results, retrieval queries, and documents can contain personal data, secrets, customer content, or protected business information. Default to metadata and secure references; capture content only for a documented purpose and permitted population.

Apply redaction before export, tenant isolation, encryption, role-based access, short retention, deletion propagation, regional controls, and audit logging. Sample based on risk and debugging need, not because storage is available.

05

Review the first wrong transition

Begin at the user-visible failure, then walk backward to the first span whose actual state diverged from the expected contract. Classify the failure before changing prompts: model, prompt, tool, retrieval, data, policy, infrastructure, or orchestration.

Escalate when a trace indicates unsafe action, cross-tenant data, repeated unknown outcomes, policy conflict, or impact beyond automated recovery. Give the human a redacted timeline, affected targets, containment status, evidence, and the decision required.

Practice activity

Instrument and review an agent trace

  1. Create a trace schema for one workflow with agent, model, retrieval, tool, approval, handoff, verification, and recovery spans.
  2. Define quality, safety, reliability, latency, token, cost, and escalation metrics with dimensions and alert thresholds.
  3. Write the telemetry data policy covering collection purpose, redaction, content capture, sampling, access, retention, deletion, and residency.
  4. Review one synthetic failed trace, identify the first wrong transition, assign a failure class, and prepare a human-escalation packet.

What to produce

  • A trace/span schema, metric catalog, service objectives, and alert rules tied to product decisions.
  • A privacy policy and trace-review artifact showing causal diagnosis, failure classification, containment status, and escalation evidence.

Reflect before continuing

Which telemetry field was useful for debugging but unjustified to retain in clear text?

Evidence

Sources and verification

Knowledge check

Make it stick.

Pass at 80%

Choose the strongest answer for each question. Your attempts become part of your device-local transcript.

01Which signal best reconstructs the causal sequence of one agent run?
02What is a better cost metric than total tokens alone?
03How should raw model and tool content be handled in telemetry?
04Where should trace diagnosis begin after a user-visible failure?
05What belongs in a human escalation packet?