# AI Red-Teaming and Safety Evals: How to Stress-Test AI Systems

> AI red-teaming tests how a system behaves under adversarial pressure: jailbreaks, prompt injection, unsafe tool use, bias, data leakage, and dangerous capability risks.

*Published 2026-07-09 · By Marcus Vance*

# AI Red-Teaming and Safety Evals: How to Stress-Test AI Systems

> In short: AI red-teaming  is adversarial evaluation for models, tools, and agentic workflows. It asks how the system fails when users, documents, tools, or attackers try to make it leak data, ignore instructions, take unsafe actions, or produce harmful output.

Capability evaluation asks whether an AI system can do the task. Red-teaming asks what happens when the task, the user, or the environment becomes hostile. For modern LLM and agentic systems, that means testing prompts, retrieved documents, tool calls, memory, policies, and human escalation paths together.

The key shift is from one-time safety review to recurring operational practice. A system that resisted last quarter’s jailbreaks can fail after a model update, a new tool permission, a fresh document source, or a prompt change. Safety is a regression surface, not a checkbox.

For leaders, the operational question is blast radius. What can the system reach, what can it change, what can it reveal, and who notices when it goes wrong? Red-teaming is valuable because it turns those abstract risks into concrete traces that engineers, lawyers, security teams, and business owners can act on.

## What does AI red-teaming test?

A serious red-team plan covers direct jailbreaks, indirect prompt injection through documents or webpages, data exfiltration, privacy leakage, toxic or discriminatory output, unsafe tool calls, over-refusal, and domain-specific harms. For agents, it also tests runaway loops, excessive permissions, irreversible actions, and whether the system can be tricked into using tools outside policy.

The  OWASP Top 10 for Large Language Model Applications  is a useful threat map because it treats LLM failure as an application-security problem, not only a model behavior problem. Prompt injection, insecure output handling, supply-chain issues, sensitive-information disclosure, and excessive agency all matter.

The strongest tests are multi-turn and system-level. A single malicious prompt is useful, but many real failures combine a user instruction, a retrieved document, a tool result, and a memory write. Red teams should test that full path because each layer can appear harmless until combined.

Table: Common AI red-team surfaces
Surface | Example attack | Control to test
--- | --- | ---
User prompt | Ignore prior instructions | Instruction hierarchy and refusal policy
Retrieved document | Hidden malicious text in a PDF | Retrieval filtering and citation checks
Tool call | Unauthorized email or database action | Least privilege and human approval
Memory | Poison future context | Memory write policy and expiration
Output | Leak secrets or unsafe advice | Output scanning and escalation

## Why are prompt injection and tool use so risky?

Prompt injection is dangerous because the model processes untrusted data and trusted instructions in the same language channel. A malicious webpage, email, or document can tell the system to reveal secrets, alter tool calls, or ignore policy. The model may not reliably distinguish instructions from data unless the surrounding application enforces that boundary.

Tool use raises the blast radius. A chatbot that says something wrong creates information risk; an agent with email, filesystem, payment, or deployment tools can create operational risk. That is why strong systems put deterministic controls around tools: allowlists, scoped credentials, spend caps, approval gates, audit logs, and rollback procedures.

The practical rule is simple: never give a model authority that the application cannot constrain. If a workflow needs production data, external messages, purchases, or deployment access, the model should request through a narrow tool with validation and human approval, not improvise with broad credentials.

## How should safety evals be measured?

Attack success rate is the basic metric: how often a class of attack causes the forbidden behavior. False refusal rate is the other side: how often the system blocks legitimate requests. Good safety programs report both because an assistant that refuses everything is safe in a narrow sense but useless in production.

The  NIST AI Risk Management Framework  frames measurement as part of a full risk lifecycle: govern, map, measure, and manage. In practice, that means red-team findings should become regression tests, policy updates, monitoring alerts, and owner-assigned remediation work rather than an isolated report.

Safety evals also need severity. A jailbreak that produces mild policy-violating text is not the same as one that leaks credentials or sends an unauthorized email. Rank findings by impact and exploitability so remediation time goes to the failures that can create real harm.

## What is the practical safety stack for agentic systems?

The practical stack is layered. Start with clear system instructions, but do not rely on them alone. Add input validation, retrieval filtering, constrained tool schemas, least-privilege credentials, approval gates for irreversible actions, output scanning, trace logging, and periodic adversarial testing. For high-risk workflows, include human review before external sends, production deploys, financial transactions, or sensitive-data disclosure.

The strongest pattern is to make unsafe behavior technically hard, not merely against policy. A model can be persuaded; an absent credential cannot. A tool that only accepts a validated schema and cannot reach production is much safer than a broad shell or API key hidden behind a polite instruction.

The final layer is monitoring. Trace every tool call, store the prompt and retrieved context where policy allows, sample live sessions, and review anomalous cost, latency, refusal, and escalation patterns. Red-teaming before launch reduces risk; monitoring after launch keeps the risk from returning quietly.

## What sources anchor this guide?

This guide uses the red-teaming and safety-evaluation corpus and anchors the operating model in OWASP application-security guidance, NIST AI risk management, and public safety-evaluation tooling.

- [OWASP Top 10 for Large Language Model Applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/) - OWASP
- [AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) - NIST
- [Inspect AI evaluation framework](https://inspect.ai-safety-institute.org.uk/) - UK AI Security Institute

## Sources

1. [OWASP Top 10 for Large Language Model Applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
2. [AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework)
3. [Inspect AI evaluation framework](https://inspect.ai-safety-institute.org.uk/)

---
Source: https://aiintelreport.com/research/research-ai-red-teaming-safety-evals
Index: https://aiintelreport.com/llms.txt · Full text: https://aiintelreport.com/llms-full.txt