Project 42Start learning
Module 5 of 7 · 58 min

Secure OpenAI Workflows with Layered Trust Boundaries

Threat-model direct and indirect prompt injection, sensitive data, excessive tool authority, and unsafe output, then enforce layered controls around OpenAI workflows.

OpenAI

By the end

You will be able to

  • Threat-model users, retrieved content, tools, data, identities, and side effects.
  • Separate trusted instructions from untrusted input without mistaking structure for enforcement.
  • Apply moderation, least privilege, human oversight, validation, and privacy controls in layers.
  • Red-team output and agent trajectories and convert failures into owned regression cases.
01

Threat-model the complete workflow

Direct injection comes from a user trying to override policy. Indirect injection can arrive in a document, web page, email, image text, memory, retrieval result, MCP response, or other tool output the model reads. Both may target secrets, data, identities, decisions, tools, or side effects.

Map actors, instruction authority, untrusted inputs, sensitive assets, tools, permissions, networks, persistence, and consequences. Model safeguards are one layer; the application remains responsible for access control and safe outcomes.

02

Separate authority and data

Keep stable policy in the appropriate instruction layer and label external content by source and trust level. Do not concatenate retrieved or tool-supplied text into privileged instructions. Preserve provenance so reviewers can distinguish evidence from commands embedded in evidence.

Delimiters and schemas reduce ambiguity but do not make content trustworthy. Minimize sensitive context, exclude credentials, apply current data controls, and validate every requested operation independently of model text.

03

Layer prevention and human oversight

Use input constraints, moderation or domain-specific classifiers, narrow tools, allowlists, output validation, rate and spend limits, sandboxing, and least-privilege credentials according to risk. Treat moderation as one signal, not a universal business-policy engine.

Require a qualified human to review consequential or high-stakes outputs and actions. Give reviewers the original evidence, uncertainty, tool trajectory, and authority to reject or escalate—not just a polished generated answer.

04

Contain data, tools, and side effects

Separate read, reversible write, communication, privileged change, and destructive authority. Bind credentials to the workload, validate tool arguments, require fresh approval when policy demands it, enforce idempotency, and verify postconditions in the system of record.

Define retention, deletion, reporting, containment, credential revocation, evidence preservation, notification, and recovery before launch. If execution may have occurred but confirmation failed, preserve an unknown-outcome state and reconcile before repeating.

05

Red-team output and trajectories

Test representative use, direct jailbreaks, poisoned retrieval, malicious tool results, data exfiltration, confused-deputy requests, approval bypass, multi-turn persistence, and misuse patterns relevant to the product. Include both expected refusals and legitimate cases that safeguards must not block.

Score the final output and the trajectory: data accessed, tools selected, arguments proposed, approvals requested, operations executed, and postconditions observed. Contain failures, repair the failed boundary, assign ownership, and add durable regression coverage.

Practice activity

Red-team and contain an OpenAI workflow

  1. Create a data-flow and trust-boundary map for one OpenAI workflow, including identities, instructions, sensitive assets, external content, tools, provider storage choices, approvals, and side effects.
  2. Write ten tests: two benign, two direct injection, two indirect injection, one data-exfiltration, one confused-deputy, one approval-bypass, and one legitimate case at risk of false blocking.
  3. Map each threat to prevention, detection, human oversight, containment, evidence preservation, and recovery controls enforced outside the model.
  4. Run the cases, record outputs and trajectories, document residual risk, and convert confirmed failures into versioned regression tests with owners.

What to produce

  • A threat model and control matrix showing enforcement owner, data lifecycle, approval boundary, detection, response, and recovery for every material threat.
  • A ten-case red-team report with trajectory evidence, false-positive review, blocked side effects, residual risks, regression cases, and accountable owners.

Reflect before continuing

If the model followed a malicious instruction perfectly, which independent control would still prevent the highest-impact harm?

Evidence

Sources and verification

Knowledge check

Make it stick.

Pass at 80%

Choose the strongest answer for each question. Your attempts become part of your device-local transcript.

01What is indirect prompt injection?
02What do delimiters or schemas guarantee?
03What is the safest role for moderation?
04Why score agent trajectories?
05What should follow a successful adversarial case?