Governing AI Agents: What Controls Organizations Need in Production

Soon after Dario Amodei, cofounder of Anthropic, recently warned that the technology industry has drastically understated the real dangers of AI, Sam Altman and Elon Musk agreed with him.  

In a rare moment of consensus among AI's most prominent leaders, all three called for the industry to slow the pace of development to give safety and oversight time to catch up. 

This agreement raises a critical question for enterprise leaders: If the people closest to the technology are sounding the alarm, shouldn't we be taking action now? And more pressingly, what does that action actually look like in practice? 

Dario suggests the answer is independent evaluators with proper access to test models and verify safety claims before deployment.  

The Enterprise Reality: AI Governance Funding Isn't the Challenge 

In Schellman's recent survey of 525 enterprise professionals on the 2026 State of AI Governance, 90% of organizations reported funding AI governance initiatives. Yet only 27% say their programs are fully mature and operationalized. That's a massive confidence-to-capability gap, and the reasons are more nuanced than simple underinvestment. 

A year ago, the constraint was getting approval for budget towards governance spending. Now, organizations have the money but lack the know-how of what to do with it. 

This manifests in dangerous ways. Anthropic recently discovered a safety filter that silently failed for multiple model generations, and nobody caught it because there was no visibility into whether the control was actually working. The filter existed but the evidence that it functioned did not, and this challenge isn't isolated.  

Most organizations treat governance controls as things that exist on paper, not systems that require active verification and measurement. 

The AI Agent Governance Paradox 

Here's where it gets complicated: organizations with mature AI governance deploy agents at 4x the rate of those with developing governance. That intuitively makes sense, as stronger governance should enable confident deployment. But a critical gap emerges: most organizations deploying agents at scale don't have defined human oversight requirements for them. 

This is because agents are being deployed faster than governance frameworks can keep pace. It's the same pattern we've seen with every technology: shadow IT, shadow AI, citizen development. The agents people know about are the mature ones; the ones without clear ownership or oversight are often invisible until they cause a problem. 

And when that happens, the response is often silence: "Whose agent is this?" "We're not sure." Without ownership, you can't have governance. 

Control Framework to Govern AI Agents 

The control framework comes down to three things:  

1. Understanding your impact (what can go wrong and what's the blast radius?) 
2. Establishing visibility (you can't secure what you can't see) 
3. Building constraints (how do you keep agents within intended boundaries at runtime, not just at design time?) 

This requires a shift from build-time governance, how agents are designed and tested before release, to runtime governance: what controls are active while they're operating in production. 

The organizations doing this right are building evidence of responsible AI practices today using frameworks like ISO 42001 and AIUC-1 to turn governance into a competitive advantage. 

If you're deploying agents, you need to start here now. Contact us to learn more about how to build your AI governance roadmap with AI agents in mind.