The Normalization of Deviance in AI · Embrace The Red

Organizations are dangerously over-relying on LLM outputs in agentic systems, treating probabilistic and potentially adversarial model responses as reliable and safe. This cultural drift mirrors the Space Shuttle Challenger disaster's normalization of deviance, where repeated success breeds complacency about known risks. Major vendors like Microsoft, OpenAI, Anthropic, and Google acknowledge security vulnerabilities including prompt injection attacks, data exfiltration, and unintended code execution, yet competitive pressure drives deployment of systems that can format hard drives, compromise databases, or act as insider threats. The absence of successful attacks is mistaken for robust security, while proper controls like sandboxing, least privilege, and human oversight are bypassed in favor of speed and automation.