Research
AI Red-Teaming and Safety Evals: How to Stress-Test AI Systems
AI red-teaming tests how a system behaves under adversarial pressure: jailbreaks, prompt injection, unsafe tool use, bias, data leakage, and dangerous capability risks.
AI red-teaming is adversarial evaluation for models, tools, and agentic workflows. It asks how the system fails when users, documents, tools, or attackers try to make it leak data, ignore instructions, take unsafe actions, or produce harmful output.
Capability evaluation asks whether an AI system can do the task. Red-teaming asks what happens when the task, the user, or the environment becomes hostile. For modern LLM and agentic systems, that means testing prompts, retrieved documents, tool calls, memory, policies, and human escalation paths together.
The key shift is from one-time safety review to recurring operational practice. A system that resisted last quarter’s jailbreaks can fail after a model update, a new tool permission, a fresh document source, or a prompt change. Safety is a regression surface, not a checkbox.
For leaders, the operational question is blast radius. What can the system reach, what can it change, what can it reveal, and who notices when it goes wrong? Red-teaming is valuable because it turns those abstract risks into concrete traces that engineers, lawyers, security teams, and business owners can act on.
What does AI red-teaming test?
A serious red-team plan covers direct jailbreaks, indirect prompt injection through documents or webpages, data exfiltration, privacy leakage, toxic or discriminatory output, unsafe tool calls, over-refusal, and domain-specific harms. For agents, it also tests runaway loops, excessive permissions, irreversible actions, and whether the system can be tricked into using tools outside policy.
The OWASP Top 10 for Large Language Model Applications is a useful threat map because it treats LLM failure as an application-security problem, not only a model behavior problem. Prompt injection, insecure output handling, supply-chain issues, sensitive-information disclosure, and excessive agency all matter.
The strongest tests are multi-turn and system-level. A single malicious prompt is useful, but many real failures combine a user instruction, a retrieved document, a tool result, and a memory write. Red teams should test that full path because each layer can appear harmless until combined.
| Surface | Example attack | Control to test |
|---|---|---|
| User prompt | Ignore prior instructions | Instruction hierarchy and refusal policy |
| Retrieved document | Hidden malicious text in a PDF | Retrieval filtering and citation checks |
| Tool call | Unauthorized email or database action | Least privilege and human approval |
| Memory | Poison future context | Memory write policy and expiration |
| Output | Leak secrets or unsafe advice | Output scanning and escalation |
Why are prompt injection and tool use so risky?
Prompt injection is dangerous because the model processes untrusted data and trusted instructions in the same language channel. A malicious webpage, email, or document can tell the system to reveal secrets, alter tool calls, or ignore policy. The model may not reliably distinguish instructions from data unless the surrounding application enforces that boundary.
Tool use raises the blast radius. A chatbot that says something wrong creates information risk; an agent with email, filesystem, payment, or deployment tools can create operational risk. That is why strong systems put deterministic controls around tools: allowlists, scoped credentials, spend caps, approval gates, audit logs, and rollback procedures.
The practical rule is simple: never give a model authority that the application cannot constrain. If a workflow needs production data, external messages, purchases, or deployment access, the model should request through a narrow tool with validation and human approval, not improvise with broad credentials.
How should safety evals be measured?
Attack success rate is the basic metric: how often a class of attack causes the forbidden behavior. False refusal rate is the other side: how often the system blocks legitimate requests. Good safety programs report both because an assistant that refuses everything is safe in a narrow sense but useless in production.
The NIST AI Risk Management Framework frames measurement as part of a full risk lifecycle: govern, map, measure, and manage. In practice, that means red-team findings should become regression tests, policy updates, monitoring alerts, and owner-assigned remediation work rather than an isolated report.
Safety evals also need severity. A jailbreak that produces mild policy-violating text is not the same as one that leaks credentials or sends an unauthorized email. Rank findings by impact and exploitability so remediation time goes to the failures that can create real harm.
What is the practical safety stack for agentic systems?
The practical stack is layered. Start with clear system instructions, but do not rely on them alone. Add input validation, retrieval filtering, constrained tool schemas, least-privilege credentials, approval gates for irreversible actions, output scanning, trace logging, and periodic adversarial testing. For high-risk workflows, include human review before external sends, production deploys, financial transactions, or sensitive-data disclosure.
The strongest pattern is to make unsafe behavior technically hard, not merely against policy. A model can be persuaded; an absent credential cannot. A tool that only accepts a validated schema and cannot reach production is much safer than a broad shell or API key hidden behind a polite instruction.
The final layer is monitoring. Trace every tool call, store the prompt and retrieved context where policy allows, sample live sessions, and review anomalous cost, latency, refusal, and escalation patterns. Red-teaming before launch reduces risk; monitoring after launch keeps the risk from returning quietly.
What sources anchor this guide?
This guide uses the red-teaming and safety-evaluation corpus and anchors the operating model in OWASP application-security guidance, NIST AI risk management, and public safety-evaluation tooling.
- OWASP Top 10 for Large Language Model Applications, OWASP.
- AI Risk Management Framework, NIST.
- Inspect AI evaluation framework, UK AI Security Institute.
Frequently asked
What is AI red-teaming?
AI red-teaming is adversarial testing of a model or AI application to discover how it fails under pressure. It includes attempts to jailbreak instructions, inject malicious content through data sources, extract secrets, produce harmful output, misuse tools, or create biased results. The output should be a prioritized list of reproducible failures and controls, not just a dramatic demo.
How is red-teaming different from normal AI evaluation?
Normal evaluation usually measures whether the system succeeds on intended tasks. Red-teaming measures whether it fails safely under hostile or unusual conditions. Both are necessary. A model may score well on helpfulness but still leak sensitive information, follow malicious document instructions, take an unsafe tool action, or fail a regulated-domain safety requirement.
What is prompt injection?
Prompt injection is an attack where untrusted text tries to override the model’s instructions. It can come directly from a user or indirectly from a webpage, email, document, or database record that the model reads. The risk is highest when the model also has tools, secrets, or memory, because a successful injection can move from bad text to bad action.
Can guardrails fully solve AI safety?
No single guardrail fully solves AI safety. Guardrails reduce risk when combined with application controls: scoped tools, content filters, retrieval policies, audit logs, human approval gates, and recurring evals. Guardrails also need monitoring because model behavior and attack methods change. In production, safety is an operating process rather than a static model setting.
How often should AI red-teaming run?
Run a baseline red-team before launch, then repeat it after major model, prompt, retrieval, or tool changes. High-risk systems should also sample live traffic, maintain a frozen adversarial regression set, and refresh attack cases periodically. Every confirmed production safety incident should become a test case so the same failure cannot quietly reappear.