Execute, Roll Back, and Close a Provider Migration

Prepare versioned routes and records

Put source and candidate behind explicit versioned adapters and configurable routing. Keep domain records, learner progress, assessment attempts, transcripts, evidence references, and stable identifiers compatible across the change; add version fields rather than silently reinterpreting old data.

Separate telemetry by provider, surface, model, adapter, prompt, policy, evaluation set, and route. Validate credentials from environment or a secret manager, least-privilege access, timeouts, retries, budget controls, data handling, and incident ownership before enabling candidate execution.

Configurable route without embedded credentials

typescript

const route = process.env.MIGRATION_ROUTE ?? "source";
const allowedRoutes = new Set(["source", "shadow", "candidate-canary"]);
if (!allowedRoutes.has(route)) throw new Error("Unsupported migration route");

const credentialsPresent = [
  process.env.ANTHROPIC_API_KEY,
  process.env.OPENAI_API_KEY,
  process.env.GEMINI_API_KEY,
].filter(Boolean).length;
if (credentialsPresent === 0) throw new Error("No provider credential is configured");

Advance through controlled stages

Start with deterministic contract tests and offline replay. Use shadow execution only when duplicate processing complies with privacy, contractual, cost, and side-effect policy; block or simulate tools so a shadow cannot repeat real actions.

Move to a small cohort or use-case canary only when offline and shadow gates pass. Cap traffic, duration, cost, retries, tool permissions, and blast radius. Expand one stage at a time after the named approver reviews critical-slice evidence.

Enforce gates and stop authority

Evaluate output quality, factuality, safety and refusal behavior, structured data, tool trajectories, state continuity, accessibility, latency, errors, usage, cost, data controls, operator load, and incident signals by critical slice. Aggregate averages must not hide a failed safety, language, accessibility, or high-value workflow.

Automate bounded stops for clear thresholds and give an on-call human authority to pause or roll back on ambiguous high-consequence signals. Preserve the evidence, route version, decision, approver, and timestamp for every hold, expansion, or rollback.

Roll back and reconcile state and side effects

Route new work back to the proven source, stop or drain candidate workers, and preserve failure evidence. Identify in-flight conversations, tool calls, file or database writes, messages, purchases, queued jobs, caches, retries, learner attempts, and records that may be incomplete or duplicated.

Use idempotency keys, operation journals, versioned state, compensating actions, and manual review queues to reconcile each side effect. Verify source health, data integrity, duplicate suppression, access cleanup, and user communication before declaring rollback complete.

Observe, close, and hand off residual risk

After promotion, keep source routing or another tested fallback until the defined observation window passes. Confirm service, quality, safety, accessibility, cost, capacity, data, support, and incident indicators; rehearse rollback again if configuration or dependencies changed.

Close only when deprecated use is removed or accepted with an owner, records and migrations are verified, dashboards and alerts are current, runbooks and training match production, credentials and permissions are right-sized, unresolved risk is accepted by the accountable owner, and the next lifecycle review is scheduled.

Practice activity

Run a failed-canary and rollback rehearsal

Create synthetic no-paid-call source and candidate traces with stable request, learner, attempt, tool-call, side-effect, and route identifiers; include one hidden critical-slice regression and one ambiguous operational signal.
Move the fixture set through contract, offline, shadow, and canary gates, recording evidence, approver, exposure, cost assumption, decision, and timestamp at each stage.
Trigger the failed canary, route new work to the source, stop or drain the candidate, and inventory every in-flight state item, tool side effect, queued action, retry, and learner record.
Reconcile the synthetic operation journal with idempotency and compensating actions, verify source health and data integrity, and measure detection, decision, rollback, and reconciliation time.
Write a closeout and residual-risk handoff that identifies production observation, lifecycle cleanup, documentation, access, owners, acceptance authority, and the next review date.

What to produce

A staged gate log and isolated trace set showing exact route versions, critical-slice results, approvals, exposure limits, and the rollback trigger.
A reconciled operation journal, recovery verification, measured rollback report, and signed closeout and residual-risk handoff.

Reflect before continuing

Which state or side effect would have remained inconsistent if rollback had only changed the traffic route?

Execute, Roll Back, and Close a Provider Migration

You will be able to

Prepare versioned routes and records

Advance through controlled stages

Enforce gates and stop authority

Roll back and reconcile state and side effects

Observe, close, and hand off residual risk

Run a failed-canary and rollback rehearsal

What to produce

Reflect before continuing

Sources and verification

Make it stick.