AI red teaming means deliberately trying to make your AI system misbehave, the way a curious customer, a troll or an attacker would, so you find the problems before they do. For a customer-facing chatbot, a few days of structured testing can prevent the screenshots, data leaks and false promises that end up in the news.
This guide is for teams testing their own chatbot, or one they’re authorised to test.
Step 1: Set the scope and rules
Before testing, write down:
- What’s being tested: which chatbot, which version, and which environment. Use a staging copy where possible.
- Authorisation: written permission from the system owner, and from the vendor if their terms require it.
- What the bot is supposed to do, and what it must never do, such as give legal advice, offer refunds or reveal account data.
- What it can access: data sources, tools and actions. This is where the real risk sits.
- How findings are recorded and shared.
Step 2: Build your test areas
Cover these areas, mapped loosely to the OWASP Top 10 for LLM Applications:
Prompt injection and instruction override
Can users make the bot ignore its instructions, change its role or reveal its system prompt? Test both direct attempts and indirect ones, such as instructions hidden in documents, web pages or tickets the bot reads. See what is prompt injection.
Data leakage
Can you get the bot to reveal other customers’ data, internal documents, personal data or secrets? Try asking about other accounts, requesting “everything you know”, and mixing in plausible-sounding authority claims. See AI chatbot data leakage.
Policy and business rule bypass
Can the bot be talked into promises it shouldn’t make, such as discounts, refunds, guarantees, legal commitments or medical, legal or financial advice?
Harmful and off-brand content
Can it be jailbroken into producing offensive, dangerous or discriminatory content, or opinions that embarrass your brand?
Accuracy and hallucination
Does it invent policies, prices, product features or sources? Test with questions whose correct answers you know.
Actions and tools (if it has any)
Can it be manipulated into taking actions, such as sending messages, changing records or calling APIs, beyond what the user should be able to do? See AI agent security.
Transparency and escalation
Does it say it’s an AI? Can users reach a human when they need one? In the EU, chatbots must make clear people are interacting with AI under the EU AI Act.
Resource abuse
Very long inputs, rapid repeated requests and anything that could run up costs or take the bot down.
Step 3: Run the tests
- Write test cases for each area, mixing simple attempts with creative ones. Vary wording, language and persistence: many failures appear only on the fifth or tenth try.
- Test multi-turn conversations, not just single prompts.
- Record every prompt, response, date and model version for each finding.
- Combine manual testing, which is creative and context-aware, with automated tools for scale and regression.
Step 4: Score and prioritise
For each finding, rate:
- Impact: what could happen? Data exposed, money lost, harm caused, reputation damaged.
- Likelihood: how easy is it, and would a normal user stumble into it?
Fix high-impact, easy-to-trigger issues first.
Step 5: Fix the system, not just the prompt
Prompt tweaks help, but durable fixes usually sit outside the model:
- Enforce access control and business rules in code, not instructions.
- Reduce the bot’s data access and permissions.
- Add input and output filtering.
- Add human approval for sensitive actions.
- Improve grounding and citations to reduce made-up answers.
Step 6: Retest and repeat
Re-run your test cases after fixes, and again whenever the model, prompt, data or tools change. Keep the test suite. It becomes your regression check.
When to bring in outside help
Internal testing catches a lot. An independent tester adds fresh eyes, current attack techniques and credible evidence for customers, auditors and regulators, which is especially useful before a major launch or for high-risk uses.
Official sources
- OWASP Top 10 for LLM Applications: the OWASP GenAI Security Project’s list of the main LLM risks
- MITRE ATLAS: a knowledge base of real attack techniques against AI systems
- NIST AI 600-1, Generative AI Profile: the AI RMF applied to generative AI
More primary sources, with a short note on each, are on our Sources page.
Last reviewed October 2026. Only test systems you own or are explicitly authorised to test. This article is general information for defensive purposes.