Red-teaming

A structured adversarial testing process in which a team deliberately attempts to make an AI system fail — by crafting inputs that cause harmful, biased, or unintended outputs. Red-teaming borrows from cybersecurity practice (where a 'red team' simulates attackers). In AI, red-teamers probe for harmful content generation, jailbreaks, prompt injection, bias amplification, and factual failures. The EU-US voluntary AI commitments and the NIST AI RMF both reference red-teaming as a baseline safety practice. Major AI labs (Anthropic, OpenAI, Google) conduct red-teaming before model releases. For small teams deploying AI, a lightweight red-teaming exercise — trying to break your own system before users do — is a practical pre-deployment step.

Before launching an AI customer service bot, a team spends two hours trying to make it give incorrect refund amounts, reveal internal pricing, or produce offensive responses. This red-teaming exercise surfaces a prompt injection vulnerability that is fixed before launch.