lesson
60 min

Red-Teaming and Safety Filters

Attack your own system on purpose, layer guardrails before and after the model, and give every block a humane fallback.

Objective

By the end of this lesson, you can plan a red-team exercise for an AI system and design layered guardrails with clear fallbacks for people who hit them.

Red-teaming

Red-teaming means trying, on purpose, to make your system fail or cause harm, so you can fix it before real users or attackers find the problem. The name comes from security and military exercises where one team plays the attacker.

A red-team plan answers:

  • Scope: which system, which version, which features.
  • Harms to probe: wrong eligibility advice, sharing private data, unsafe advice in a crisis, biased treatment, prompt injection, offensive output.
  • Who attacks: include people from the communities you serve. They know the questions and phrasing your team will miss.
  • Rules: use made-up data only; stop and report anything that touches real people.
  • What happens next: every finding gets an owner, a fix, and a new test case.

Probe in every language you serve, with slang and misspellings, through long conversations, and with uploaded documents.

A national example

Layered guardrails

A guardrail is a check that blocks or changes input or output. No single guardrail is enough. Layer them:

  1. Input checks: detect crisis language, block obvious abuse, enforce length limits, flag likely injection (knowing it will miss some).
  2. Retrieval limits: only search documents the user may see.
  3. System prompt rules: scope, refusals, handoffs.
  4. Output checks: schema validation, citation checks, PII detection, safety classifiers for harmful content.
  5. Action gates: human approval for anything that writes, sends, or spends (Module 5).
  6. Monitoring: watch block rates and sample blocked items (Module 10).

Every block needs a fallback

People who hit a guardrail still need help. A blank screen or "I can't help with that" fails them twice. Design the fallback:

  • Crisis language: show crisis resources immediately (for example, the 988 Suicide and Crisis Lifeline in the U.S.) and offer a person. Never block silently.
  • Out of scope: say what the assistant can do and how to reach staff.
  • Safety filter false positive: a kind message and a path to a person. Log it so you can tune the filter.

Track false positives (good requests blocked) and false negatives (bad requests allowed). A filter that blocks too much will drive away the people you serve, often those who phrase things differently.

Red-team and guardrail plan

0 of 8 done

Go deeper: Cybersecurity, Plain and Simple: A Practical Playbook for Owners and Operators

Ask. Answer. Build together.