AI safety

AI safety, in the applied sense, is the set of defenses a deployed model sits behind after training is already finished — input filters, output filters, red-teaming, and runtime monitoring — built on the assumption that alignment training alone will not catch every harmful request a live system encounters. OpenAI ran GPT-4 through more than 50 external red-teaming experts across cybersecurity, bioweapons, and misinformation domains before its March 2023 release, and still shipped the model behind an additional moderation layer rather than relying on training alone; Carnegie Mellon researchers published the GCG attack in July 2023, showing that an appended adversarial suffix of nonsense tokens could reliably jailbreak aligned models from multiple vendors, which is exactly the kind of post-training failure deployment-layer defenses exist to catch. The pipeline a production system runs on every request looks like this.

Red-teaming before release and monitoring after release are the same discipline at different times. Anthropic's Responsible Scaling Policy, published in September 2023, formalizes this as AI Safety Levels (ASL) — ASL-2 covers today's deployed models with standard safeguards, ASL-3 requires stronger security and misuse-resistance commitments tied to specific dangerous-capability thresholds (such as uplift toward bioweapons synthesis), and the policy commits Anthropic to pausing further scaling if evaluations cross a threshold before matching safeguards exist. OpenAI's Preparedness Framework, also 2023, runs the same idea with its own named risk categories — cybersecurity, chemical/biological/nuclear/radiological, persuasion, and model autonomy — each scored and gated before a model ships.

The field now has shared standards instead of each lab inventing its own checklist. NIST published its AI Risk Management Framework in January 2023; OWASP released its Top 10 for Large Language Model Applications the same year, cataloguing prompt injection as its number-one risk category; the EU AI Act, which entered into force in August 2024, imposes binding risk-tiered obligations on providers of general-purpose AI models above a compute threshold. These give an external auditor something concrete to check a deployment against, rather than taking a vendor's safety claims on faith.

Red-teaming has itself been automated. Rather than only paying domain experts to hand-craft attacks, labs now run an adversarial model against the target model — a "red-LLM" generates attack prompts, a safety classifier scores the target's response, and the loop repeats at a scale no team of human red-teamers could match, with the automated findings used to prioritize where human experts spend their time. No defense in this stack is complete alone: a classifier trained on known jailbreak patterns misses genuinely novel ones, which is why red-teaming, guardrails, and monitoring are treated as a layered system rather than any single control a team can point to and call "the" safety feature.