Updating the taxonomy of failure modes in agentic AI systems: What a year of red teaming taught us

We're reaching an inflection point where AI agents have stopped being a research curiosity and become operational infrastructure. The question is no longer whether they'll be deployed—they already are—but whether we understand the risks well enough to deploy them safely.

The past year has given us a crash course in that problem. A full year of red teaming against agentic systems has surfaced a meaningful shift in how we should categorize failure modes. We're not talking about hallucinations or prompt injection in the old sense anymore. When an AI agent runs autonomously across multiple steps, maintaining state and calling external tools, the failure surface becomes fundamentally different. An error in step three compounds into step five. A tool call made without proper validation cascades. I find this distinction important because it changes where we focus defensive effort.

Meanwhile, the practical deployment is accelerating hard. Meta, Google Cloud and IBM, NVIDIA—the infrastructure players are all racing to make agents easier to run at scale. Arena's research shows the adoption pattern is clear: people use agents most heavily in professional contexts, especially in tech. That makes sense. A developer can supervise an agent writing code or querying a database in ways a non-technical user cannot. But supervision and control weaken as agents become more autonomous and long-running. NVIDIA's work on efficiency for multi-turn agents is genuinely useful, but it's also greasing the wheels for deployment before we've fully solved the safety question.

By the way, Anthropic's recent warning about self-improvement deserves more than a dismissive eye-roll. The company is flagging a specific concern: models approaching the capability threshold where they could improve themselves without explicit human intervention. Whether that's imminent or still years away depends on your model of capability progression, but it's not paranoia to think about it now. The fusion researchers at Princeton just demonstrated something interesting in the other direction—using machine learning to prevent instabilities in tokamaks at commercial scale. That's agents working as intended: augmenting human expertise in a domain where the stakes are physical and measurable.

The tension is real. We need agents deployed to solve actual problems—fusion energy, drug discovery, complex logistics. We also need to genuinely understand what breaks when they fail. The red teaming is helping with the latter, but it's a race against the rate of deployment. I'm watching whether the safety work keeps pace or becomes an afterthought once these systems are too embedded to pause.