Project 42Start learning
Module 5 of 7 · 58 min

Secure Gemini Workflows with Layered Trust Boundaries

Threat-model prompt injection, unsafe content, sensitive data, excessive tool authority, and factuality failures, then enforce layered controls around Gemini.

Google

By the end

You will be able to

  • Threat-model direct and indirect injection across users, multimodal content, retrieval, tools, and state.
  • Separate instruction authority from untrusted data and minimize sensitive context.
  • Use Gemini safety signals as one layer alongside external authorization, validation, oversight, and recovery.
  • Red-team output and tool trajectories and convert failures into owned regression cases.
01

Threat-model the complete Gemini system

Direct injection comes from a user trying to override policy. Indirect injection can arrive in documents, images, audio, video, web results, memory, MCP content, or function responses. Both may target secrets, identity, data, decisions, tools, or side effects.

Map actors, instruction authority, untrusted inputs, modalities, sensitive assets, tools, credentials, persistence, and consequences. Built-in model protections are one layer; the application owner remains responsible for system safety and factual use.

02

Separate instructions, data, and modalities

Keep stable policy in the supported system-instruction boundary and label external content by source, modality, and trust level. Preserve provenance and never promote document or tool text into privileged instructions merely because the model can read it.

Tags and schemas reduce ambiguity but do not create trust. Minimize sensitive content, exclude credentials, apply current data controls, validate claims and tool proposals, and enforce identity and resource authorization outside Gemini.

03

Use safety signals in layers

Gemini API safety settings and built-in protections can produce prompt feedback, safety ratings, blocked content, and safety finish outcomes. Configure adjustable settings from a documented use-case policy and current terms, then test both harmful cases and legitimate cases at risk of false blocking.

Do not treat filter probability as a severity assessment or authorization decision. Combine safety signals with input constraints, domain safeguards, output validation, least privilege, budgets, informed human review, user reporting, monitoring, and incident response.

04

Contain tools, data, and side effects

Separate reads, reversible writes, communications, privileged changes, and destructive actions. Use narrow tools, workload-scoped credentials, fresh approval where required, operation identifiers, idempotency, sandboxing, and verified postconditions.

Define retention, deletion, containment, credential revocation, evidence preservation, notification, and recovery before launch. If a side effect may have occurred but confirmation failed, retain an outcome-unknown state and reconcile before another action.

05

Red-team outputs and trajectories

Test representative tasks, direct jailbreaks, poisoned multimodal input, malicious retrieval and function results, data exfiltration, confused-deputy requests, approval bypass, repeated-turn attacks, factuality failures, and legitimate sensitive use cases.

Score final output and trajectory: inputs consumed, tools proposed, arguments, approvals, operations executed, and postconditions. Contain confirmed failures, repair the failed boundary, assess false positives, assign owners, and add versioned regression coverage.

Practice activity

Red-team and contain a Gemini workflow

  1. Create a multimodal data-flow and trust-boundary map including identities, instructions, data, model settings, safety signals, tools, approvals, storage, and side effects.
  2. Create ten synthetic cases: benign, direct and indirect injection, poisoned modality or tool output, exfiltration, confused deputy, approval bypass, factuality failure, and legitimate content at risk of false blocking.
  3. Map every threat to prevention, detection, informed oversight, containment, evidence, and recovery controls enforced outside the model.
  4. Run the no-paid-call fixtures, record output and trajectory evidence, document residual risk, and add confirmed failures to regression coverage.

What to produce

  • A threat and control matrix identifying enforcement owner, data lifecycle, safety-setting rationale, approval boundary, detection, response, and recovery.
  • A ten-case fixture report with trajectories, safety signals, false-positive review, blocked side effects, residual risks, regressions, and owners.

Reflect before continuing

If Gemini followed a malicious instruction perfectly, which independent control would still prevent the highest-impact harm?

Evidence

Sources and verification

Knowledge check

Make it stick.

Pass at 80%

Choose the strongest answer for each question. Your attempts become part of your device-local transcript.

01What is indirect prompt injection?
02What do tags and schemas guarantee?
03How should Gemini safety ratings be used?
04Why score tool trajectories?
05What follows a successful adversarial case?