Evaluate Agent Behavior with Evidence
Build representative, adversarial, and regression evaluation sets that measure outcomes, trajectories, safety, cost, and latency before agent changes reach production.
By the end
You will be able to
- Turn product requirements and failure history into versioned evaluation cases.
- Score final outcomes, tool trajectories, policy compliance, cost, and latency with explicit rubrics.
- Combine deterministic checks, human review, and model graders without treating any grader as ground truth.
- Compare a candidate with a baseline and make a documented release decision.
Begin with a claim the evaluation can test
An evaluation is a repeatable experiment, not a demo. Name the workflow version, target population, expected behavior, unacceptable behavior, scoring rule, sample, environment, and release threshold before seeing candidate results.
Separate task success from style, safety, policy, efficiency, and user experience. A single average can hide a severe regression in one critical slice.
caseId: [STABLE ID]
slice: [NORMAL | EDGE | ADVERSARIAL | INCIDENT]
input: [VERSIONED FIXTURE]
expectedOutcome: [OBSERVABLE RESULT]
allowedTrajectory: [TOOLS + BOUNDS]
forbidden: [POLICY OR SAFETY FAILURE]
rubric: [CRITERIA + SCALE + EVIDENCE]
budget: [LATENCY + COST + TURNS]Cover reality, boundaries, and prior failures
Sample common tasks, important user groups, difficult long-tail cases, languages and modalities you support, and the data distributions the application will actually see. Preserve a holdout set so repeated prompt editing does not overfit every known case.
Add prompt injection, ambiguous intent, missing permissions, stale retrieval, tool timeout, conflicting evidence, unsafe requests, and every confirmed production incident. Synthetic cases can expand coverage, but a human should review whether they represent the product risk.
Score what happened and how it happened
Use deterministic graders for schema, exact constraints, calculations, tool permissions, postconditions, and latency or cost budgets. Use a rubric and blinded human review when quality depends on judgment. Calibrate model graders against human-labeled examples and monitor disagreement.
For agents, final-answer quality is insufficient. Inspect whether the correct tools were selected, arguments and approvals were valid, retries were bounded, evidence was preserved, and the run stopped in the right terminal state.
Compare candidates on identical cases
Run the baseline and candidate against the same versioned cases, environment, budgets, and grader definitions. Report per-slice pass rates, critical failures, score distributions, latency percentiles, cost, and uncertainty instead of only a global mean.
Define must-not-regress slices and minimum practical improvement before the run. Investigate changed cases individually and require human sign-off when a candidate alters safety, permissions, or high-impact behavior.
Turn production evidence into the next offline suite
Sample privacy-reviewed production traces, explicit user feedback, escalations, and incidents into a governed failure corpus. Remove or transform sensitive content, preserve the behavior that caused the failure, and obtain the required consent and retention controls.
Version cases, fixtures, rubrics, graders, model and tool configurations, and results. A score without those inputs cannot support reproducibility or a rollback decision.
Practice activity
Build and run an agent release evaluation
- Choose one agent workflow and write ten versioned cases: four representative, two boundary, two adversarial, and two prior-failure or incident cases.
- Create a 100-point rubric plus deterministic checks for schema, permissions, postconditions, turns, latency, and cost.
- Blind-score a baseline and candidate, record grader disagreement, and report results by slice rather than only average.
- Make ship, hold, or rollback decisions using thresholds written before the comparison, then add one failed case to the regression set.
What to produce
- A ten-case evaluation set, 100-point rubric, deterministic grader definitions, and version manifest.
- A baseline-versus-candidate report with slice metrics, critical failures, grader disagreement, cost and latency, and a signed release decision.
Reflect before continuing
Which aggregate score initially hid the failure that mattered most to the release decision?
Evidence
Sources and verification
- Evaluation best practicesOpenAI · verified 2026-07-25
- Using the Evaluation ToolAnthropic · verified 2026-07-25
- Why evaluate agentsGoogle · verified 2026-07-25
- Artificial Intelligence Risk Management FrameworkNIST · verified 2026-07-25
Knowledge check
Make it stick.
Choose the strongest answer for each question. Your attempts become part of your device-local transcript.