State of the field
Federal policy changed. EO 14110 was rescinded in January 2025. OMB M-25-21 and M-25-22, both dated 3 April 2025, are the current federal-use and acquisition memoranda listed by OMB. Their scope is federal agencies and acquisition; they are not a general private-sector red-teaming mandate.
Anthropic’s results illustrate why benchmark scope matters. Its pre-release bounty, separate public demo, and automated synthetic evaluation used different tests, systems, and conditions. Reported outcomes from one do not establish a model-wide guarantee or a production result for another application.[3]
AI red teaming has moved from informal probing toward more structured adversarial testing. Scope, attack set, grader, model version, system prompt, and defense configuration all affect what a test can establish.
Practice varies
Microsoft has published lessons from testing more than 100 generative-AI products, while Anthropic has published separate pre-release, public-demo, and synthetic-evaluation results. Their methods and outcomes are not directly interchangeable, which is why every result on this page names its test boundary.[2][3]
Major lab efforts
Microsoft AI Red Team
Formed 2018 · 100+ products tested
One of the first red teams to cover both security and responsible AI. Published a white paper in January 2025 setting out 8 key lessons, and open-sourced the PyRIT framework for automated attack orchestration.[2]
Anthropic
Frontier Red Team · Constitutional AI
Published automated red-teaming work using model-vs-model loops and scoped evaluations of Constitutional Classifiers. The reported results depend on the named test, target behavior, and system configuration.[3]
NVIDIA
Garak LLM Vulnerability Scanner
Garak is an open-source LLM vulnerability scanner with extensible probes and detectors. Treat feature counts, maintainership, provider support, and compatibility as release-specific; check the current repository before selecting it for a test.[4]
Large-scale public testing
| Event | Scale | Key Finding |
|---|---|---|
| DEFCON 2023 | 2,244 hackers, 17,000+ conversations, 8 LLMs | 21 harm categories tested across Anthropic, OpenAI, Google, Meta |
| Crucible Platform | 214,271 attempts, 1,674 users, 400 days | Average user: 128 attack attempts across 4 challenges |
| Anthropic public demo | 339 jailbreakers, 300,000+ interactions | One universal jailbreak reported |
| Anthropic pre-release bounty | 183 participants, 3,000+ hours, 2 months | No universal jailbreak found in that scoped test |
How to read attack-rate studies
Attack Success Rate (ASR) depends on the target behavior, attack set, model version, application context, grader, and defenses. A paper or provider evaluation is evidence about that named test, not a current vulnerability rating for a model family or a deployed application.
Benchmark discipline
Record the exact model and configuration, attack corpus, target behaviors, scoring rule, grader, exclusions, retries, and date. Re-run the evaluation against the deployed system; do not import a published percentage as a production baseline.
Older studies can inform test design, but their percentages are not durable product facts.
With defensive measures
Provider and third-party evaluations can show whether a named defense improved results under a defined protocol. Validate the same control in the deployed application and preserve the test scope, failures, and residual risk.
Key Insight
Model-side alignment can reduce some attacks without eliminating application-level risk. Use layered controls, test shared dependencies, and treat model and control effectiveness as deployment-specific.
Attack-technique coverage
Include varied techniques and multi-turn paths in the local test plan. Their relative success is system-specific and should be measured against the deployed application rather than inferred from a static cross-model table.
| Technique | Test focus | Mechanism |
|---|---|---|
| Roleplay/Persona | Authority, persona, and fictional framing | Impersonation, fictional characters, hypothetical scenarios |
| Logic Traps | Conditional and contradictory instructions | Conditional structures, moral dilemmas, contradictions |
| Encoding Tricks | Every encoding accepted by the input path | Base64, zero-width characters, leetspeak, ROT13 |
| Multi-Turn Human | State carried across turns and tools | Gradual escalation across conversation turns |
| Automated Single-Turn | Repeatable regression coverage, not a substitute for human exploration | Against well-defended models with safety layers |
Multi-Turn Gap
Human exploration and automated regression testing exercise different failure paths. Use both, document their scope, and do not treat one benchmark as a complete estimate of deployed risk.
Attack taxonomy
Prompt-Level Attacks
Data Extraction Attacks
Safety Bypass Attacks
Capability Abuse Attacks
Microsoft’s 8 key lessons from 100+ products
In January 2025, Microsoft’s AI Red Team published its white paper detailing lessons from testing over 100 generative AI products since 2018. These findings represent the most extensive industry experience with structured AI adversarial testing.[2]
System-Level Attacks Win
Relatively simple attacks targeting weaknesses in end-to-end systems are more likely to succeed than complex algorithms targeting only the underlying AI model. Red teams should adopt a system-wide perspective.
Red Teaming ≠ Benchmarking
Benchmarks measure preexisting notions of harm on curated datasets. Red teaming explores unfamiliar scenarios and helps define novel harm categories. Both are necessary but serve different purposes.
Human Judgment Remains Essential
Despite automation benefits, human judgment is essential for prioritizing risks, designing system-level attacks, and assessing nuanced harms. Many risks require subject matter expertise, cultural understanding, and emotional intelligence.
Simple Attacks Often Work
Attackers often use simple, practical methods like hand-crafted prompts and fuzzing to exploit weaknesses. Sophisticated academic attacks are less common in practice than straightforward exploitation.
Mental Health Matters
Organizations need to consider red team members’ mental health, since they “may be exposed to disproportionate amounts of unsettling and disturbing AI-generated content.” Support structures are essential.
Security is Never Complete
AI models amplify existing security risks and create new ones. Theoretical research shows that for any output with non-zero probability, a sufficiently long prompt exists to elicit it. The goal is raising attack cost, not elimination.
Use AI as Force Multiplier
AI-generated attacks can lack creativity or context understanding. Use AI to brute-force simple variations while human experts analyze and guide the process. Completely hands-off AI red teaming isn’t yet viable.
Document with TTPs
Use a structured ontology to model attacks including adversarial actors, TTPs (Tactics, Techniques, and Procedures), system weaknesses, and downstream impacts. This enables systematic tracking and improvement.
Red teaming tools
Microsoft’s PyRIT and NVIDIA’s Garak are open-source tools used for AI red teaming, not formal industry standards. Confirm current maintenance, releases, compatibility, and provider integrations before adoption.
Microsoft PyRIT
Python Risk Identification Tool
Microsoft AI Red Team: “Enabled a major shift from fully manual probing to red teaming supported by automation”
NVIDIA Garak
LLM Vulnerability Scanner
NVIDIA: “Similar to nmap or Metasploit Framework, garak does comparable things for LLMs”
Tool comparison
| Capability | PyRIT | Garak |
|---|---|---|
| Attack orchestration | ✓ Extensive | ✓ Good |
| Pre-built attack library | ✓ TAP, PAIR, Crescendo | ✓ Extensible probe library |
| Multi-turn attacks | ✓ Native support | ◐ Limited |
| Multimodal support | ✓ Images, audio | ✓ Images |
| Reporting | ✓ JSON, scoring | ✓ Detailed remediation |
| Best for | Complex orchestrated attacks | Broad vulnerability scanning |
Defense effectiveness
Scoped research can show improvement from a named defense under a defined protocol. Validate that control in the deployed application; no single technique provides complete protection.
Constitutional Classifiers (Anthropic)
Anthropic reported improvement in a named synthetic evaluation and published separate pre-release and public testing. Treat each as scoped evidence about that protocol; re-test the deployed application and do not reuse the reported rate as a current production score.
Defense-in-depth layers
Block known attack patterns, encoding tricks, and malicious payloads before they reach the model.
RLHF, Constitutional AI, and other alignment techniques that train refusal behavior into the model.
Constitutional Classifiers and similar systems that detect and block harmful outputs before delivery.
Rate limiting, user authentication, capability restrictions, and audit logging at the application layer.
Human review for high-risk actions, escalation procedures, and continuous monitoring of system behavior.
Regulatory requirements
Red-teaming obligations are role- and system-specific. EU AI Act Article 55 addresses adversarial testing for providers of GPAI models with systemic risk; high-risk AI systems have separate risk-management, testing, and conformity duties. NIST AI RMF and its GenAI profile are voluntary guidance unless another instrument incorporates them.
| Regulation | Requirement | Effective | Penalty |
|---|---|---|---|
| EU AI Act (Art. 55) | Adversarial testing for providers of GPAI models with systemic risk | Role-specific | Use current Article 99 analysis |
| EU AI Act (High-Risk) | Risk management, testing, and conformity duties under provisions separate from Article 55 | Dec 2027 or Aug 2028 for relevant Omnibus pathways | Provision- and role-specific |
| NIST AI RMF | Voluntary risk-management guidance; testing should reflect context and risk | Now | Voluntary framework |
| NIST 600-1 GenAI | Specific GenAI risk mitigations including red teaming | July 2024 | Framework |
| Colorado SB 26-189 | ADMT notices, disclosures, correction, review, and records; no NIST safe harbor | Jan 1, 2027 | Current statutory analysis required |
NIST Adversarial ML Guidance
In January 2024, NIST published guidance identifying four specific types of AI cyberattacks with recommended mitigations:[8]
Corrupting training data to manipulate model behavior
Exploiting legitimate data access for unauthorized purposes
Extracting sensitive information from models
Crafting inputs that bypass detection or classification
Building a red team program
Based on research from Microsoft, Anthropic, and academic studies, effective AI red team programs share common characteristics. The SEI notes that no standardized protocols yet exist for generative AI red teaming, so organizations have to build their own.[9]
Team composition
Security Expertise
Traditional security testing skills, threat modeling, attack methodology
AI/ML Knowledge
Understanding of model behavior, training dynamics, alignment techniques
Domain Expertise
Knowledge of your specific use case, regulatory requirements, user context
Creative Thinking
Ability to find unexpected attack paths that automated tools miss
Testing cadence
Consider a scoped pre-deployment red team where the threat model and applicable assurance requirements justify it; this is not a universal legal mandate
Red team when capabilities change significantly or new features added
Test when underlying models are updated, because model changes can introduce new vulnerabilities
Choose a risk-based periodic cadence; “quarterly” is an example, not a universal regulatory requirement
Where useful, automate selected regression tests for known patterns using tools such as PyRIT or Garak
Regression Testing Gap
Feffer et al. (2024) found that fewer than 1/3 of 42 enterprise AI programs track post-fix regression. Many organizations fix vulnerabilities but don’t verify the fixes or check for reintroduction in later updates.[10]
Five-phase red-team assessment record
This five-phase approach synthesizes published testing practices into a structured assessment record. The resulting material can support review against selected controls, but is not by itself compliance evidence or a legal determination.
Illustrative testing sequence
Scoping
Define what you’re testing and establish boundaries aligned with regulatory requirements.
- Identify AI systems in scope (models, applications, integrations)
- Map to regulatory requirements (EU AI Act risk level, NIST functions)
- Define threat actors and attack motivations
- Establish rules of engagement and access levels
Deliverable: Scoping document with regulatory mapping, signed authorization
Threat Modeling
Map the attack surface using Microsoft’s TTP ontology and prioritize testing areas.
- Document AI system architecture and data flows
- Map attack taxonomy categories to system components
- Prioritize attacks based on likelihood and impact
- Cover system-level attacks as well as model-level ones (per Microsoft lessons)
Deliverable: Threat model document, prioritized attack plan
Attack Execution
Execute structured attacks combining automated tools with manual human testing.
- Run scoped automated scans with a current compatible Garak release
- Execute multi-turn attacks with PyRIT orchestration
- Conduct manual creative testing alongside automated regression coverage
- Document all findings with reproduction steps
Deliverable: Attack logs, finding documentation, severity ratings
Reporting
Document findings with actionable remediation guidance and an explicit evidence boundary.
- Create detailed finding reports with CVSS-style severity ratings
- Map findings to regulatory requirements (EU AI Act articles, NIST categories)
- Recommend specific remediations with defense layer mapping
- Generate executive summary for board/customer communication
Deliverable: Technical report, executive summary, and scoped testing-evidence package
Verification
Confirm remediations are effective and establish regression testing baseline.
- Retest all findings after remediation
- Test for regressions and new findings within the defined retest scope; this cannot establish that no new vulnerabilities exist
- Create automated regression test suite for CI/CD integration
- Document closure evidence for compliance records
Deliverable: Verification report, regression test suite, compliance closure documentation
Apply the playbook where your AI already acts
Bring the system boundary, testing objective, and evidence question. We’ll discuss where red-team findings meet operational controls and bounded records.