The Red-Team Engagement Playbook: Independent Adversarial Testing for LLM Apps and Agents

An LLM application that reads documents, calls tools and acts on behalf of users has an attack surface unlike any traditional software. Inputs are natural language, behavior is probabilistic, and the line between data and instructions is thin. Independent red teaming is how organizations find out what an attacker, or an ordinary user having an unusual day, can make the system do. This playbook explains how to scope, run and report an engagement that produces findings engineers can fix and leaders can act on.

Why independence matters

The people who build a system know how it is supposed to work, and that knowledge shapes the tests they think to run. NIST's Generative AI Profile (NIST AI 600-1) addresses this directly, recommending that organizations "verify those conducting structured human feedback exercises are not directly involved in system development tasks for the same GAI model"¹. The same profile calls for engaging internal and external evaluations and red-teaming with people who are familiar with the context of use¹.

Practitioners reach the same conclusion. Microsoft's AI Red Team, which has operated since 2018, published lessons from red teaming more than 100 generative AI products. Among them: "You don't have to compute gradients to break an AI system," "AI red teaming is not safety benchmarking," and "The human element of AI red teaming is crucial"²³. In other words, simple, creative attacks from people who think differently from the build team often matter more than sophisticated tooling.

A red team's value comes from asking the questions the build team did not think to ask, then proving the answer with a reproducible test.

Independence also supports governance. A report produced by a team outside the delivery chain carries more weight with risk committees, auditors and regulators than a self-assessment, and it gives the build team a clear, impartial list of what to fix. Quantabase provides this through Assure, which evaluates and red-teams AI systems independently of the teams that built them.

The regulatory and standards context

Adversarial testing has moved from good practice to an explicit expectation in several frameworks:

• EU AI Act, Article 55. Providers of general-purpose AI models with systemic risk must "perform model evaluation in accordance with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing of the model." These obligations have applied since 2 August 2025⁴.
• EU AI Act, Article 15. High-risk AI systems must achieve "an appropriate level of accuracy, robustness, and cybersecurity" throughout their lifecycle, with technical measures addressing data poisoning, model poisoning, adversarial examples or model evasion, confidentiality attacks and model flaws⁵. Under the Digital Omnibus on AI, which entered into force on 27 July 2026, the application date for stand-alone Annex III high-risk systems moved to 2 December 2027, and to 2 August 2028 for AI embedded in products covered by EU product safety legislation⁶.
• NIST AI Risk Management Framework. AI RMF 1.0 (January 2023) organizes risk work into Govern, Map, Measure and Manage, and the Generative AI Profile (July 2024) lists twelve risk areas including confabulation, data privacy, information security and value chain and component integration⁷¹.
• CISA and UK NCSC. The joint Guidelines for Secure AI System Development, developed with 21 other agencies and ministries including all G7 members, cover secure design, development, deployment, and operation and maintenance⁸.
• UK AI Security Institute. The institute released Inspect, an open-source evaluation framework under the MIT License, in May 2024 to support standardized testing of model capabilities and safety⁹.
• OWASP Top 10 for LLM Applications (2025). The most widely used taxonomy of LLM application risks, from LLM01 Prompt Injection to LLM10 Unbounded Consumption¹⁰.

Scoping checklist

A well-scoped engagement is the single largest predictor of useful findings. OpenAI's published approach to external red teaming identifies four design decisions: the composition of the red team, the model or system versions testers can access, the interfaces and instructions provided, and how the data is synthesized into evaluations¹¹. The checklist below turns those decisions into concrete questions.

1. System description. What does the application do, for whom, and in which workflow? Which model or models does it use, and which versions?
2. Architecture. System prompt, retrieval sources, memory, tools and APIs, output destinations (screens, emails, databases, code execution).
3. Trust boundaries. Which inputs come from users, which from third parties (web pages, emails, uploaded files), and which from internal systems?
4. Agency. What can the system do without a human approving it? Which actions are reversible?
5. Data sensitivity. What personal, confidential or regulated data can the system see, and what should it never reveal?
6. Harm policy. What content and behaviors are out of bounds for this product and domain?
7. Environment and access. Production, staging or a dedicated test environment; test accounts with realistic roles; rate limits and cost caps.
8. Rules of engagement. Testing windows, prohibited actions, data handling, stop conditions and the contact for urgent findings.
9. Success criteria. What the engagement must answer, such as "Can an external email cause the agent to send data outside the organization?"

Who must be involved

Diversity of perspective is part of the method. NIST recommends interdisciplinary teams spanning a wide range of competencies, domain expertise and lived experience¹, and Anthropic has described domain-specific expert testing, multilingual and multicultural testing, and automated testing as complementary methods¹².

Threat modeling

Threat modeling turns the scope into a target list. For each component, ask who could influence it, what they could gain and what the system would do in response.

1. Draw the data flow. Map every input source, the model calls, every tool and every output destination.
2. Mark trust boundaries. Any place where untrusted text enters the context window is a potential injection point.
3. List assets. Customer data, credentials, system prompts, financial actions, reputation.
4. Identify adversaries. Malicious users, external content authors, compromised insiders and curious users who stumble into edge cases.
5. Prioritize scenarios. Rank by impact and plausibility; spend the most effort where an agent can take irreversible actions on sensitive data.

Attack categories

Direct prompt injection

The user instructs the model to ignore its rules, adopt a new role or reveal its system prompt. OWASP ranks prompt injection first in its 2025 list (LLM01) and treats system prompt leakage as a separate risk (LLM07)¹⁰. Tests include instruction overrides, role-play framing, encoding and obfuscation, and multi-turn escalation.

Indirect prompt injection

Instructions are planted in content the system retrieves: a web page, a shared document, an email or a support ticket. Greshake and colleagues showed in 2023 that LLM-integrated applications "blur the line between data and instructions," enabling remote attacks including data theft and information ecosystem contamination without direct access to the interface¹³. For agents and retrieval-augmented systems, this is usually the most important category to test.

Data exfiltration

The attacker aims to extract sensitive information, either from the model's context (other users' data, retrieved documents, credentials) or through a side channel such as a rendered link or image that encodes data in a URL. This maps to OWASP LLM02 Sensitive Information Disclosure and LLM05 Improper Output Handling¹⁰.

Tool misuse and excessive agency

The attacker steers the system into calling tools in unintended ways: sending messages, modifying records, running code or spending money. OWASP LLM06 Excessive Agency describes damaging actions performed in response to unexpected, ambiguous or manipulated model outputs, with root causes in excessive functionality, excessive permissions and excessive autonomy¹⁸. Tests check whether permissions are scoped, whether high-impact actions require confirmation and whether tool outputs are themselves treated as untrusted.

Jailbreaks

Attempts to bypass the model's safety behavior to produce prohibited content. Microsoft's red team notes that simple techniques are often effective, which is why manual creativity remains essential alongside automation².

Domain harms

Failures specific to the use case: incorrect financial or medical guidance, unfair treatment of customer groups, fabricated citations, or commitments the business cannot honor. These align with OWASP LLM09 Misinformation and NIST's confabulation and harmful bias risks¹⁰¹.

Resource abuse

Inputs designed to drive excessive token use, long tool loops or costly calls, covered by OWASP LLM10 Unbounded Consumption¹⁰.

Tooling

Open-source tools help testers cover more ground and make results repeatable. They complement, rather than replace, human testing.

A practical pattern is to use automated tools for breadth in the first days, then direct human effort at the system's specific tools, data and workflows, where the most consequential findings usually sit.

Severity rubric

Severity combines impact and exploitability. A consistent rubric lets system owners prioritize and lets leadership compare results across engagements.

Because model outputs vary, record the success rate for each finding (for example, 7 of 10 attempts) rather than a single pass or fail.

Reproduction-step format

Each finding should allow an engineer who was not present to reproduce it. A consistent template:

1. Finding ID and title. Short and specific.
2. Severity and category. Using the rubric above and the OWASP LLM Top 10 identifier.
3. Environment. Application version, model and version, configuration, user role, date and time.
4. Preconditions. Data, documents or account state required.
5. Steps. Numbered inputs exactly as submitted, including any planted content.
6. Observed result. Model output, tool calls made and side effects, with logs or screenshots.
7. Expected result. What the system should have done.
8. Success rate. Successful attempts over total attempts.
9. Recommended mitigation. Specific controls, for example narrowing tool permissions or adding confirmation for external sends.

Reporting template

Retesting cadence

A red-team result is a snapshot. OpenAI notes that red teaming "captures risks at a specific moment, which may change as models evolve"¹¹, and Microsoft's eighth lesson is that "the work of securing AI systems will never be complete"². A sustainable cadence has four layers:

1. Fix verification. Retest every critical and high finding as soon as a fix is deployed.
2. Regression suite. Convert each confirmed finding into an automated test that runs in the deployment pipeline.
3. Change-triggered testing. Run a focused engagement whenever the model version, system prompt, tools or data sources change materially.
4. Periodic full engagement. Repeat a full independent test on a regular schedule, typically quarterly or semiannually for systems with agency over sensitive data.

Retesting matters more as systems gain autonomy. Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls¹⁷. Regular, independent adversarial testing is one of the most direct ways to keep risk controls in step with capability.