Most enterprise AI deployments today aren’t fully autonomous. There’s usually a human in the loop. The agent proposes a plan, chooses which tools to call, and surfaces its reasoning or actions for approval before executing.
That works today because the number of decisions is still manageable. But as agents become more capable, asking a human to validate every tool call, every API request, and every execution-time decision quickly stops being practical. At some point, the human becomes the bottleneck, and you’ve lost the benefit of having an autonomous system in the first place.
That’s where I think the real governance problem starts.
Gartner recently predicted that by 2027, 40% of enterprises will demote or decommission autonomous AI agents because governance gaps won’t become apparent until after production incidents. I don’t think we have to wait for full autonomy to run into those problems.
If you’ve used Claude Code, you’ve already experienced the problem
You just probably haven’t thought about it as an AI agent governance problem.
You give Claude Code a task, and a minute later it’s read half your codebase, called a handful of tools, maybe spun up a sub-agent, edited a file, hit an API, and somehow found a config file you forgot existed. Most of that happened before you had a chance to follow along.
That’s not because Claude Code is doing anything wrong. It’s because that’s how capable AI agents work. They gather context, adapt their plan, choose which tools to call, and make decisions based on what they learn while they’re executing.
What struck me is that almost all of those decisions happened after I gave the agent the task.
That’s the governance problem.
Today, most AI agent governance is front-loaded
Most teams still focus on governing what an AI agent starts with, even though the most important decisions happen after execution begins. That model works well for deterministic software because the execution path is largely known in advance. AI agents are different. They gather new context, adapt their plan, and continuously make decisions while they're running.
The natural instinct is to apply least privilege, just as we do for humans. Give an agent the minimum access it needs. Scope the tools it can use. Issue short-lived credentials. That’s still good security, and it isn’t going away.
But least privilege isn’t the whole story anymore. It tells you what an agent could access, not whether a specific action should happen in the context of the task it’s executing.
An agent might have permission to GitHub, Jira, and Salesforce. The harder question is whether it should update this issue, modify this repository, or read this customer record while acting on behalf of this user.
That’s a runtime decision.
Right now, most teams think about AI agent governance in three layers.
Tier 1: Static governance. Acceptable use policies. Approved model lists. Compliance documents legal signed off on. These define what agents are supposed to do. They say nothing about what agents actually do.
Tier 2: Structural governance. Tool allowlists. MCP connectors. Sandboxed APIs. Better than documentation because you’ve narrowed the blast radius, but it’s still a configuration decision made before the agent starts.
Tier 3: Instructional governance. CLAUDE.md. AGENTS.md. .cursor/rules. System prompts. The agent harness. “Never access production.” “Always ask before deleting.” This is where most sophisticated teams are today.
The problem is that all three layers stop before execution.
Once the agent starts reasoning, choosing tools, and interacting with systems, the first three layers have already done their job. They shape the agent’s behavior, but they don’t enforce it.
That’s why I think there’s a fourth layer.
Tier 4: Runtime governance. Instead of evaluating policy once at setup, policy is evaluated at the moment of every tool call. Before the agent touches an external system, the runtime has the context of the task, the identity of the user the agent is acting for, the requested action, and the applicable policies. It can make a deterministic decision: allow or deny.
That's the point where governance becomes enforcement instead of guidance.
Before the agent touches anything
What runtime governance of AI agents actually looks like
The easiest way I’ve found to think about it is to start with how we already govern people, then ask what changes when the actor is an AI agent.
The same principles still apply: identity, authorization, and access. The difference is that those decisions now have to account for who delegated the task, what the agent is doing, and what it’s trying to access at that moment.
Identity - Who is this agent?
Agents don’t look like users. They spin up, perform a task, call a few tools, and disappear. Runtime governance starts by giving every agent a real identity that’s tied to who delegated the task, what it’s doing, and what scope it was given. Without that, everything that follows becomes difficult to attribute.
Runtime authorization - Should this action be allowed?
The second capability is runtime authorization.
Every tool call becomes a policy decision. Before an agent touches an external system, something has to decide whether that specific action should be allowed for this user, through this agent, on this resource, in the context of the task it’s currently executing.
What’s interesting is that the industry seems to have independently converged on where that decision belongs. Claude Code, Codex, Gemini CLI, and Microsoft’s Agent Governance Toolkit all expose hooks that let developers inspect or intervene while an agent is running.
The enforcement point already exists.
The challenge is making those decisions with enough context to determine whether the action is actually authorized.
Access - How does the agent get the permissions it needs?
The final capability is access.
Long-lived API keys, service accounts, and .env files don’t fit well with autonomous agents. Instead, access should be granted just in time, scoped to the specific action being performed, and expire as soon as that work is complete.
Taken together, that’s what runtime governance looks like: identity that can attribute every action, authorization that evaluates every tool call in context, and access that’s granted only for the work the agent is performing.
The Emerging AI Agent Governance Stack
One thing that stood out to me while researching this space is that nobody is building AI agent governance as a single product.
Instead, the market is emerging one layer at a time. Companies are building different parts of what increasingly looks like an AI agent runtime governance stack: agent identity, runtime policy evaluation, access & credentials, and audit. Once you look at the landscape through that lens, it becomes much easier to understand.
What’s also interesting is that the market is splitting into two camps. Some companies are building agent-native infrastructure from scratch. Others are extending the identity and security infrastructure enterprises already rely on.
Agent Identity
Identity is where I think the market gets interesting. There seem to be two approaches emerging.
The incumbents already own human identity. Okta, Microsoft Entra, Auth0, Ping Identity, and others aren't going away. If enterprises are going to deploy millions of AI agents, it makes sense that those platforms will eventually need to understand agents alongside people. The question is whether agent identity becomes another object inside existing IAM systems, or whether it requires an entirely new identity layer.
Keycard is the clearest example. Rather than treating an agent as another service account, it gives agents first-class identities tied to the human who delegated the task, the workload they’re executing, and the scope they were granted. It federates with existing IAM systems instead of replacing them, then adds delegation, workload attestation, and short-lived credentials on top.
There’s also an adjacent category that’s becoming increasingly important: non-human identity (NHI) governance.
Companies like Oasis Security aren’t creating identities for new AI agents. They’re helping enterprises discover and govern the machine identities they already have-service accounts, OAuth applications, API keys, and other credentials that autonomous agents increasingly inherit. As AI agents become another consumer of machine identities, NHI governance starts looking like a foundational piece of the runtime stack.
Runtime Policy
Once an agent has an identity, every tool call becomes a policy decision.
This is where companies like Cerbos and Oso fit.
Cerbos approaches the problem as an authorization policy engine. Rather than embedding permission logic throughout application code, it evaluates every authorization request against declarative policies at runtime. For AI agents, that means every tool call can be evaluated independently before it's allowed to execute.
Oso approaches the problem from the application authorization side. It helps developers define fine-grained authorization based on roles, relationships, and context, making it possible to evaluate what an AI agent should be allowed to do on behalf of a user instead of simply inheriting all of that user's permissions.
Microsoft’s open-source Agent Governance Toolkit is another interesting signal. Rather than inventing a new runtime, Microsoft is extending existing agent frameworks with runtime policy enforcement. To me, that’s a sign they see runtime policy evaluation becoming foundational infrastructure rather than a product differentiator.
Access & Credentials
Identity answers who.
Policy answers whether.
Something still has to answer how an AI agent obtains access.
That’s where companies like Arcade and Nango fit, although they’re solving different parts of the problem.
Arcade focuses on delegated, just-in-time access for AI agents. Instead of giving an agent a long-lived API key or service account, it obtains a short-lived credential on behalf of the user, scopes it to the specific action, and expires it when the work is complete.
Nango approaches the problem from the OAuth and API integration layer. It provides infrastructure for OAuth flows, token refresh, and credential management across hundreds of third-party APIs while allowing organizations to retain control of their own credential infrastructure.
Both are moving away from standing credentials toward credentials that exist only for the duration of the work an AI agent is performing.
Audit
Audit is less of a standalone product category than an outcome of the other three working together.
If identity, policy, and credentials are all evaluated at runtime, an audit trail naturally falls out of the system. You know who delegated the task, which policy evaluated the request, what credential was issued, and every action the agent took.
Companies like Keycard, Cerbos, and Arcade all contribute different parts of that picture - identity, policy decisions, and execution events, but together they make it possible to reconstruct what happened after the fact.
That’s the interesting part to me.
These companies aren’t building competing products. They’re building different layers of what looks increasingly like the runtime stack for AI agents.
Executive Order 14409 Signals Where AI Agent Governance Is Headed
On June 2, 2026, President Trump signed Executive Order 14409. Most of the attention went to Section 3, which targets frontier model developers.
I think Section 4 is the part enterprises should be paying attention to.
It directs the Attorney General to prioritize enforcement of existing laws, including the Computer Fraud and Abuse Act (CFAA), against people who use AI to gain unauthorized access to computer systems. It explicitly calls out AI agents.
The Executive Order doesn’t create a new legal standard for enterprises.
What it does signal is that authorization is becoming the question.
Not:
Did we tell the agent what to do?
But:
Can we prove this action was authorized?
To me, that's where AI agent governance is headed. Not more prompts or policy documents, but runtime identity, authorization, access control, and auditability that can answer that question every time an AI agent takes an action.