1. Regulatory intelligence - know the rules. Ingest, track, and interpret regulations and regulatory change, then turn them into structured, monitored obligations.

2. Compliance decisioning - decide the case. Apply external regulations, internal policies, and prior decisions to a customer, transaction, document, communication, vendor, or questionnaire.

3. Compliance operations - execute the workflow. Gather evidence, route cases, capture approvals, file, remediate, and preserve an audit trail.

The three layers also reflect how the market has developed. Regulatory intelligence is the oldest and most mature category; regtech companies have sold regulatory data and change-management software for more than a decade. Decisioning is where many AI-native startups begin because case-level review is bounded, measurable, and well suited to language models. Operations is the current frontier, as agents become capable of handling variable, unstructured, multi-step work across documents and systems.

Autonomous execution is starting to appear in specific, lower-risk workflows, but broad autonomy barely exists. The constraint is partly model reliability and partly whether institutions, regulators, and customers are willing to let a machine make and act on a consequential decision.

I think that value and defensibility generally increase as a product owns more of the decision-to-action loop. A company does not need to begin with regulatory intelligence or move through every layer in order. But products that move from providing information to owning decisions, execution, and governed authority capture more of the workflow, more institutional memory, and more control over how compliance work gets done.

1. Why compliance AI is suddenly investable

’s Everything, Everywhere Is Compliance, published by James da Costa and Angela Strange in May 2026, makes the macro case. The United States has more than 400,000 compliance officers representing over $40 billion in annual labor spend. The work is also becoming harder to staff, with annual churn above 20% while regulatory complexity continues to grow.

The market was already large. What changed is how much of the work software can perform. Vision-language models can interpret documents that traditional OCR could only extract, while frontier models have improved at reading rules, applying them to facts, and identifying exceptions. Computer-use and longer-horizon agents extend this beyond review, allowing systems to carry context across documents, databases, and legacy applications. Benchmarks are not proof of production reliability, but they show that the addressable workflow is expanding.

The buyer calculus is changing as well. Faster KYC improves onboarding, faster marketing review increases speed to market, and better investigations reduce backlogs. Compliance software is no longer sold only as a way to reduce headcount or avoid fines. YC’s Fall 2026 Requests for Startups reflects that shift, calling for AI-native infrastructure that replaces fragmented tools and manual workflows.

a16z describes compliance as regulation, the software trying to codify it, and the people connecting the two. The three layers in this piece describe which part of that human work a product automates; the autonomy axis describes how much responsibility moves to the system.

2. The three layers of compliance AI

How AI moves from interpreting regulation to executing compliance work

Layer 1 — Regulatory intelligence: know the rules

Regulatory intelligence turns a constantly changing body of regulation into something a compliance team and eventually an agent can act on. The workflow begins with horizon scanning but continues through a consolidated rules inventory, applicability analysis, obligation extraction, and mapping to internal policies, procedures, controls, risks, and owners. Provenance has to remain intact throughout, because the company needs to know where an obligation came from, which version was in force, and what changed.

Regulatory source → obligation → policy → procedure → control

This is the oldest and most mature layer. Regtech companies have collected, classified, and distributed regulatory information for more than a decade, so the market already has established buyers, large incumbents, and recognizable product categories.

It is also consolidating. CUBE acquired Thomson Reuters Regulatory Intelligence and Oden in 2024. The combined company now covers more than 750 jurisdictions and employs close to 250 regulatory subject-matter experts and legal and compliance professionals alongside its AI products. Other established players include Wolters Kluwer, Corlytics, in which Verdane made a majority investment in 2024, and Compliance.ai, which Archer acquired in the same year.

Bloomberg Industry Group’s acquisition of Regology in June 2026 follows the same logic. Bloomberg combined its legal, tax, and government information with software that monitors regulatory change, determines which rules apply to a particular company, and connects those changes to internal policies, risks, and controls.

AI changes this layer by turning regulatory content into company-specific work, moving the product from identifying the rules toward deciding how they apply.

Ascent is a good example. Its platform creates company-specific obligation inventories and maps them to internal policies, procedures, and controls. Its acquisition of Waymark in 2024 combined horizon scanning with obligations management, extending the product from detecting regulatory change to showing where that change matters inside the institution.

Layer 2 — Compliance decisioning: decide the case

Compliance decisioning applies external regulations, internal policies, and prior decisions to a specific customer, transaction, document, communication, vendor, or questionnaire. The system evaluates the facts and produces a determination or draft supported by evidence, rationale, and citations. Where the answer is uncertain or the risk exceeds a defined threshold, it escalates the case to a human reviewer.

Case facts + rules + policy + precedent → determination + rationale + evidence

The distinction from regulatory intelligence is simple. Regulatory intelligence determines what the business must do in general; compliance decisioning determines what it should do in this case. Its central value is replacing hours spent searching policies, source documents, evidence, and prior decisions with a grounded first determination that a reviewer can verify.

This is a natural starting point for AI-native companies because the work can be bounded and evaluated. A buyer can compare the system’s conclusions with those of experienced reviewers, measure reductions in review time, and test whether similar cases produce consistent results. The product does not initially need authority to approve or file anything. It can create value by completing the first pass and sending exceptions to the appropriate reviewer.

I would draw the line here: a decisioning product applies governed rules, company policy, or precedent to a specific set of facts and produces an answer that can be checked against a human reviewer’s. That distinguishes compliance decisioning from generic document generation or retrieval.

This decisioning pattern is appearing across several compliance workflows.

Marketing and communications review. Companies including Norm Ai, Sedric, and Hadrius translate requirements such as UDAAP, SEC and FINRA rules, regulatory guidance, and internal company policy into review criteria that can be applied to financial-services marketing. The systems review advertisements, emails, websites, social posts, and other customer communications for prohibited claims, missing disclosures, unsupported statements, and policy violations.

Life sciences has a specialized version of the same workflow called medical, legal, and regulatory review. Veeva’s acquisition of Copli and launch of Falcon MLR in June 2026 brought agentic review of promotional and medical content into Veeva’s existing life-sciences platform. The acquisition validates marketing review as a decisioning wedge, but it also shows the strategic risk: an incumbent that already owns the content and approval workflow can absorb the review product.

DDQs, RFPs, and security questionnaires. Products such as 1up, Arphie, and Tribble generate answers from approved company materials, policies, prior responses, and supporting evidence. The value is not simply writing prose faster. It is determining which source is authoritative, retrieving previously approved language, attaching evidence to each answer, and routing low-confidence responses to the person qualified to approve them.

Third-party risk. Coverbase analyzes questionnaires, contracts, and third-party evidence against a company’s controls, surfaces exceptions, and produces a risk assessment. Its broader platform also handles vendor intake, evidence collection, follow-ups, continuous monitoring, and remediation. The risk determination sits in decisioning, while the surrounding vendor lifecycle pushes the product into compliance operations.

KYC, KYB, and financial-crime review. Arva AI applies an institution’s policies across customer and business onboarding, sanctions screening, and transaction-monitoring alerts. It assembles the relevant identity, ownership, transaction, and screening data, produces a risk determination supported by evidence, and routes uncertain or higher-risk cases to human investigators.

Parcha initially offered a similar product focused on automating KYC, KYB, and compliance reviews for banks and fintechs. It has since retired that product line and relaunched as Grep AI, applying the agent architecture it developed in financial-services compliance to a broader set of research-intensive, high-stakes workflows.

At the same time, the identity platforms underneath these products are moving into the same decisioning and workflow layer. Persona, which raised $200 million at a $2 billion valuation, has expanded from identity verification into automated decisions, follow-ups, workflows, and case management.

I think this is the more important takeaway from Parcha’s pivot. The KYC and KYB platforms underneath standalone AI reviewers are expanding into decisioning and workflows themselves, leaving less room to build a durable moat at the review layer. The larger opportunity is to own the case, the workflow, or proprietary data or to take the underlying agent architecture into a broader market, as Parcha did.

Product and trade compliance. Trava and Complir use the same underlying architecture: translate a body of regulation into machine-readable logic, map it to product data, and make a product-level compliance determination.

Audit and regulatory submissions. Denki collects evidence, tests controls, and produces traceable workpapers for internal-audit and compliance programs. Ritivel turns clinical data and prior submissions into reproducible, source-linked drafts for life-sciences filings. Both begin with decisioning but move into operations as they assemble the evidence and produce the auditor- or regulator-ready document.

The durable asset in decisioning is not access to the model. It is the system built around it: the mapping between regulations and company policy, the evidence supporting each conclusion, domain-specific evaluations, and the institutional memory created as reviewers accept, reject, or modify recommendations. Capturing why a reviewer overrode the system is especially important, because that is how individual judgments become reusable precedent rather than disappearing into an audit log.

Decisioning is therefore both an attractive entry point and an unstable place to stop. It creates measurable value without requiring the product to act autonomously, but it becomes more embedded when it also gathers evidence, routes exceptions, manages approvals, and records the final outcome. That is the natural path from decisioning into compliance operations.

Layer 3 — Compliance operations: execute the workflow

Compliance operations automates the work that compliance analysts and operations teams perform to move a case from intake to closure. This includes gathering evidence, researching across systems, requesting missing information, routing the case, capturing approvals, preparing filings, managing remediation, and preserving an audit trail. Decisioning provides the judgment within the case; operations carries the case through the governed workflow.

Intake → evidence → decision → approval or action → audit record

Workflow automation is not new. Traditional systems can move a case through a predefined sequence, but they generally depend on structured inputs and explicit rules for every step. When a document is incomplete, an answer is ambiguous, or the next action depends on information scattered across several systems, a person has historically carried the context and determined what to do next.

Agents expand the kind of work software can perform. They can read unstructured documents, retrieve information from multiple systems, determine which evidence is missing, adapt the next step, and operate the applications through which compliance work gets completed. The workflow still needs defined permissions, approval gates, and escalation rules, but it no longer has to anticipate every possible path in advance.

Several companies illustrate how this layer is developing.

Financial-crime operations. Unit21 began as a fraud and AML platform and is adding agents inside the system where alerts, investigations, and regulatory filings already live. Its agents gather evidence, investigate cases against the institution’s procedures, recommend dispositions, draft regulator-ready narratives, and produce a transparent log of the data accessed and steps performed. The institution continues to define its risk appetite, thresholds, approval requirements, and escalation paths.

SOX testing. Petual imports a company’s risk and control matrix, maps evidence to the relevant controls and samples, executes the prescribed testing procedures, and generates audit-ready workpapers. Deviations are surfaced for review and remediation. The determination whether the control operated effectively is decisioning; collecting the evidence, performing the test, documenting the result, and moving exceptions toward resolution are compliance operations.

Financial licensing. Brico manages the operational work surrounding state financial licenses, including applications, renewals, periodic reports, amendments, and status tracking. Licensing is a particularly clear operations workflow because the work extends far beyond determining which licenses are required. Companies must assemble information from across the business, complete forms, track jurisdiction-specific deadlines, respond to deficiencies, and maintain each license after approval.

There are two routes into compliance operations: incumbents can add agents to their existing systems of record, while agent-first companies can absorb the workflow, integrations, and case data around them. a16z frames the buyer’s choice as keeping the incumbent as a backend, rebuilding internally, or buying an AI-native replacement. YC similarly points toward AI-native products that consolidate fragmented compliance tools.

I think the two routes converge. Incumbents will add agents, while agent-first companies will build toward systems of record. Both want to own not only the final decision but how the work was performed: the evidence, policy, exceptions, approvals, and outcome. That execution history becomes institutional memory and makes the product the place where compliance work is executed, governed, and remembered.

3. The trust stack: what gives AI permission to act

The three layers describe what a compliance product does. The trust stack describes what it needs to do that work reliably, safely, and within the institution’s authority. The more judgment, workflow, and consequential action a product assumes, the more of this infrastructure it needs.

Understanding the work

• Document competence: Read PDFs, Word documents, spreadsheets, screenshots, IDs, filings, questionnaires, and marketing assets without losing the structure or context that controls the decision.

• Company context and master-data mapping: Connect regulations, evidence, and decisions to the company’s legal entities, customers, products, accounts, jurisdictions, controls, owners, and systems. Without that mapping, the system may understand the rule but apply it to the wrong part of the business.

Grounding the decision

• Traceability: Tie every material conclusion to the evidence, internal policy, and regulatory language supporting it. The system needs to show not only which sources it retrieved, but why they governed the decision.

• Institutional memory: Preserve prior filings, decisions, approved and rejected language, exceptions, reversals, and precedent. That history must remain linked to the policies and rules in effect at the time.

• Auditability: Record who or what decided, when, under which policy and model version, and any human review or override.

Governing the system

• Permissions and information barriers: Treat access as a first-class control. The system should retrieve information, expose cases, and use tools only within the boundaries of the user, entity, jurisdiction, and task.

• Authority: Distinguish between permission to read, recommend, escalate, approve, file, reclassify, contact a customer, and change a system of record. Each action requires its own authorization boundary.

• Human control: Support review, correction, rejection, and escalation when evidence is incomplete, policy is ambiguous, risk is high, or human judgment is required by law or internal policy.

Executing the workflow

• Workflow orchestration: Sequence the work, route cases, manage handoffs, enforce approval gates, and escalate exceptions from intake through final disposition. Historically, this connective tissue was a person carrying context between documents, inboxes, databases, and applications.

• Operational competence: Perform the concrete actions inside the workflow-redline a document, populate a template, preserve formatting, request missing information, upload evidence, submit a filing, export an artifact, and write the outcome back to the correct system.

A regulatory-intelligence product may only need a subset of these capabilities. A decisioning product needs most of them in at least a basic form to produce reliable case-level judgments. An operations product and especially one acting with limited human involvement needs the full trust stack because its mistakes can propagate into customer communications, filings, approvals, and systems of record.

The trust stack converts model capability into operational permission. It is what allows a compliance product to move from advising to doing, and eventually to acting within defined limits.

4. The autonomy frontier

Autonomous compliance closes the loop. The system detects an issue, gathers context, makes a determination, takes an authorized action, records its rationale, and updates the relevant systems of record. Humans define the guardrails and supervise exceptions rather than executing every routine case.

Detect → investigate → decide → act → record → monitor

This is different from rules-based automation, where the trigger, logic, and response are specified in advance. An autonomous agent can navigate a variable path, determine what evidence it needs, choose which tools to use, and resolve ambiguity. The hard part is not giving it the ability to act; it is making sure it stops when it lacks the evidence, confidence, or authority to continue.

Unit21 and Chartis Research’s agentic AI maturity spectrum frames autonomy as a progression:

1. Rules-based automation: Predefined conditions trigger deterministic workflows.

2. AI-assisted: The system recommends actions, but humans perform the work and decide.

3. Agentic with human oversight: Agents execute the workflow, while humans approve material decisions.

4. Autonomous within governed limits: The system completes permitted decisions and actions, while humans supervise exceptions and quality.

These levels are configured by workflow, jurisdiction, risk tier, and action. A system might close a low-risk false positive, recommend an outcome on a medium-risk case, and require explicit approval before rejecting a customer or submitting a regulatory filing. “Human in the loop” is not one setting; it is a set of controls around individual actions.

The constraint is both technical and institutional. The system has to remain reliable, recognize incomplete evidence, and produce a defensible record of what it did. The institution has to determine what can be delegated, scope the agent’s authority, and assign accountability when it errs. Autonomy will therefore arrive workflow by workflow, and the ability to act safely and prove why the action was appropriate may become one of the deepest moats in compliance.

5. Where founders should start

The best entry point is usually a narrow, expensive workflow with a visible output and a human process against which the product can be evaluated.

Regulatory intelligence works as a starting point when rules change frequently across jurisdictions and the product can propagate those changes into policies, controls, risks, and owners. A better summarizer is not enough in a mature and consolidating market. The product needs to determine applicability and connect regulatory change to company-specific work.

Compliance decisioning works when there is a high-volume review task with a clear human benchmark: marketing review, KYC analysis, alert triage, medical review, product classification, vendor assessment, or due-diligence questionnaires. Decisioning makes it relatively easy to demonstrate value, but it is also easier for an incumbent to bundle if the startup does not capture the surrounding evidence, precedent, or workflow.

Compliance operations works when the process spans multiple systems and produces a durable artifact: an investigation, filing, license, audit workpaper, approval record, or completed onboarding case. This is where a startup has the clearest path to owning the system of record for the work, because the product captures not only the result but the evidence, approvals, exceptions, and execution history behind it.

Autonomy should be a direction, not a day-one claim. A company earns the right to automate consequential actions workflow by workflow, after proving reliability and building the necessary permissions, evaluation systems, escalation paths, and audit controls. The strongest initial use cases are usually routine, high-volume decisions with clear policy boundaries and a safe path for escalation.

The founder test is simple:

Does the product only tell the customer what is compliant, or is it becoming the place where compliant work is executed, governed, and remembered?

The first can be a valuable tool. The second has a stronger path to becoming infrastructure.

6. From compliance answers to compliance actions

Not every company needs to span all three layers. Regulatory-intelligence and decisioning products can remain valuable on their own, but a company selling an agentic future needs to own more of the loop: the workflow, evidence, permissions, decision history, and eventually the authority to act.

The more of that loop a product owns, the harder it becomes to replace. Most compliance AI today helps companies determine what is compliant; the next generation will make the work compliant, and eventually keep it that way.

The playbook was simple: pick a frontier model, call the API, and build your product around whatever latency, pricing, throughput, and rate limits the provider offered. That wasn’t because companies didn’t care about inference. It was because they were trying to build useful products as quickly as possible. Foundation model APIs made that incredibly easy. They handled the infrastructure, gave developers instant access to frontier models, and continuously upgraded those models over time. If paying a little more for inference meant shipping months faster, it was an easy trade.

That tradeoff is starting to break.

We’re entering the second wave: own and optimize.

As AI products mature, inference is no longer something you simply consume - it’s becoming something you optimize. Not just because it’s expensive, but because inference is becoming an application-specific systems problem. A voice AI company, a coding copilot, a research agent, and a document processing pipeline are all calling an LLM, but they have completely different latency, throughput, concurrency, and cost requirements. The default API has to optimize for everyone. Your application doesn’t.

That naturally raises the question: why now?

What changed over the last two or three years that suddenly made inference something companies want to own instead of simply consume?

It wasn’t one breakthrough. It was a handful of shifts that all happened at roughly the same time.

What Changed?

The forces reshaping how companies consume inference

01 — The Workload Changed

From Conversational to Agentic

Most AI applications up until this point were conversational or workflow-based. A user would submit a prompt, maybe the application would make a couple of LLM calls behind the scenes, and then return an answer. Teams spent most of their time improving prompts, adding examples, tweaking outputs, or chaining together a handful of predefined steps. Token usage was relatively predictable because the application controlled the workflow.

Agentic systems are different.

A single user prompt is no longer a single request. It’s the beginning of an execution loop. The model reasons, plans, decides whether it needs to use tools, retrieves context, calls those tools, evaluates the results, and repeats until the task is complete. If you’ve used Claude Code or Codex, you’ve already seen this. You ask it to fix a bug, and behind the scenes it searches your codebase, opens files, writes code, runs tests, retries when something fails, and keeps going until it has a solution.

Figure 1. A chat interaction usually ends after one or two model calls. An agent loops through planning, tool use, evaluation, and retries until the task is complete.

The important shift is simple: inference no longer scales with prompts. It scales with the amount of work an agent performs on your behalf. Every reasoning step, tool call, retry, or planning step can trigger another inference request. Some tasks might finish in a handful of model calls. Others can require hundreds.

Figure 2. The economics change quickly. What was pennies for a chat session becomes meaningful infrastructure spend once agents continuously execute work.

The surprising part is where those tokens actually go.

Most people assume they’re paying for the final answer. In reality, a large portion of the inference bill is everything required to get there: conversation history being carried forward, system prompts, tool definitions, planning, retries, and intermediate reasoning. As workloads become more agentic, the overhead around the answer grows alongside the answer itself.

Figure 3. In an agentic workflow, much of the inference budget is spent on orchestration rather than the final user-visible output.

02 — The Constraints Changed

The workload changed. The optimization target changed with it.

Once companies started building agents instead of simple chatbots, they realized there isn’t one definition of “good” inference anymore. Every application is trying to optimize for a different combination of latency, throughput, and cost, which means there isn’t a single inference stack that’s optimal for everyone.

Underneath, every inference request has two phases. The first is prefill-processing the input prompt before the model can start generating a response. This determines Time to First Token (TTFT), or how long the user waits before seeing the first word. The second is decode-generating the response one token at a time. This determines Time Per Output Token (TPOT), or how quickly the rest of the response streams back. Most optimizations in the serving stack are trying to improve one or find a better balance between the two, without blowing up cost.

Figure 4. Different AI applications optimize for different combinations of latency, throughput, and cost. There isn’t a universally optimal inference stack, only one that’s optimal for the workload.

This is where the tradeoffs become obvious. A voice AI company wants the model speaking in a couple hundred milliseconds because every delay feels awkward. A coding copilot also cares deeply about latency, but it has to maintain that experience across thousands of concurrent users. A research agent running overnight is happy to wait a few extra minutes if it cuts inference costs in half. A batch document processing pipeline doesn’t care about latency at all, it wants maximum throughput and GPU utilization.

They’re all building agentic applications. They’re all running inference. But they’re solving completely different optimization problems.

That’s why the default API stops being enough. Foundation model providers have to optimize for millions of developers across millions of different workloads. Your application only has one workload. The closer inference becomes to your product, the more those default tradeoffs stop making sense.

03 — Open source caught the frontier

DeepSeek, Qwen, Llama, Kimi K2, GLM- the open ecosystem has matured quickly. There isn’t one “best” model anymore. Different models lead on different benchmarks and different workloads. Some are better at reasoning, others at coding, multilingual tasks, or long-context retrieval. That gives companies something they didn’t really have two years ago: choice.

Where is inference actually being optimized?

By this point, it’s clear inference is no longer just an API call. It’s an optimization problem. So where is all of this innovation actually happening?

The answer is across the entire stack. Some companies are rebuilding hardware. Others are rebuilding the software layer sitting on top of it. Both matter, but they operate on very different time horizons.

Hardware (Brief, But Honest)

Inference isn’t just changing software. It’s changing hardware too.

Training and inference stress hardware differently. Training is dominated by massive parallel computation. Inference is dominated by latency, memory bandwidth, and serving requests efficiently. That’s why we’re seeing a new generation of inference-first silicon alongside GPUs.

Three themes stand out.

First, specialized inference chips are becoming real. Companies like Groq, Cerebras, and d-Matrix proved there was room for architectures optimized specifically for inference rather than general-purpose GPU workloads. Whether those companies remain independent matters less than the fact that the industry has validated the need for inference-first hardware.

Second, hardware and models are starting to be designed together. Google has been doing this with TPUs for years, and OpenAI's reported Jalapeño effort points in the same direction. General-purpose GPUs have to support every workload. Custom silicon only has to optimize for one. The competitive advantage is no longer just the chip or the model, it's how well the entire stack works together.

Third, inference is pushing hardware toward full-stack systems. NVIDIA’s answer is Rubin. It isn’t just a new GPU, it’s an entire rack designed as one system, combining CPUs, GPUs, networking, interconnects, and software. That’s because inference isn’t just a compute problem anymore. Models like DeepSeek and Qwen3 constantly move tokens between experts running on different GPUs. The chip still matters, but so does everything connecting those chips together. The competitive advantage is no longer the accelerator. It’s the system around it.

Purpose-built inference hardware will absolutely matter.

But hardware refresh cycles happen over years. Software improvements ship every week.

That’s why the rest of this article focuses on the software stack. Today, it’s where companies are finding the biggest improvements in cost, latency, and throughput.

Software Stack

Inference isn’t one optimization problem anymore. It’s a stack of them.

Every layer exists because it’s solving a different bottleneck. Some make models portable. Others squeeze more work out of GPUs. Others manage deployments or optimize agent runtimes. Together, they determine the latency, throughput, and cost of every inference request.

Here’s how I think about the software stack, starting closest to the hardware and moving upward toward the application.

Layer 1 — Model Runtime
Purpose: Portable execution
Solves: Deployment fragmentation

This layer sits closest to the hardware. One of the messiest parts of AI today is that every piece of hardware wants your model in a different format. NVIDIA has one optimization stack. Apple has another. TPUs have another. CPUs have another. The model might stay the same, but the deployment pipeline changes every time.

Muna
Compiles Python inference code into optimized native executables, removing much of the runtime overhead and deployment complexity. Instead of shipping Python, containers, and a long chain of dependencies, you deploy a lightweight binary that starts faster and runs efficiently across different hardware.

Modular
Takes a broader approach by rebuilding the runtime itself. Think of MAX as the engine that runs AI models and Mojo as the language developers use to build AI software. Together, they’re trying to hide hardware-specific complexity so the same AI software can execute efficiently across CPUs, GPUs, and other accelerators.

Layer 2 — Model Serving Engine
Purpose: Maximize GPU utilization
Solves: GPU underutilization

This is where inference economics are won or lost.

Running a model efficiently is much harder than just loading it onto a GPU. Requests arrive at different times, prompts have different lengths, outputs finish at different times, and GPUs end up spending more time waiting than doing useful work. The job of the serving layer is to keep that hardware as busy as possible.

That’s where techniques like continuous batching, disaggregated prefill, and custom kernels (e.g. paged attention) come in.

Instead of serving one request at a time, they continuously pack new requests onto the GPU, reuse memory more efficiently, and dramatically improve utilization. The model doesn’t change. The hardware doesn’t change. You simply get more tokens out of the same GPU, which means lower cost and higher throughput.

Inferact
vLLM has become the default serving engine for production LLM deployments. If you’re serving chatbots, copilots, or general-purpose AI applications at scale, chances are you’re either using vLLM or evaluating it. Inferact is building the commercial layer around that ecosystem.

RadixArk
SGLang started with efficient structured generation but has quickly become a favorite for agentic and RL workloads, where models are reasoning, using tools, and running complex execution loops. RadixArk is commercializing that next generation of serving infrastructure.

Both companies were founded by many of the researchers behind the underlying open-source projects. It’s a familiar playbook: build the infrastructure in the open, then build the company around operating it at production scale.

Market signal: Both companies are betting that the serving layer becomes valuable in the same way Red Hat built a business around Linux - by commercializing critical open infrastructure rather than owning it.

Layer 3 — Deployment Platform
Purpose: Run inference reliably in production
Solves: Production complexity.

Getting a model to run efficiently is only half the problem. You still have to deploy it, scale it, monitor it, and keep it running as traffic changes. That’s the job of the deployment layer. These companies sit one layer above the serving engine, handling GPU provisioning, autoscaling, deployments, routing, observability, and everything else needed to operate inference in production.

Baseten
Built for production inference. Models stay warm, serve traffic continuously, and come with the operational tooling you’d expect from a production service. If your model is powering a customer-facing application with real latency and uptime requirements, this is the type of platform you build on.

Modal
Built for on-demand compute. Instead of keeping GPUs running 24/7, infrastructure spins up when your code runs and disappears when it’s done. It’s a much better fit for batch jobs, fine-tuning, parallel workloads, and agents that don’t need persistent endpoints.

The interesting part is that even though both abstract away GPU infrastructure, they just optimize for completely different workload patterns. Baseten assumes your model is always serving users. Modal assumes compute should exist only when there’s work to do.

Market signal: The line between deployment platforms and managed inference clouds is already starting to blur. Companies like Baseten, Fireworks, and Together are expanding vertically across the stack, increasingly bundling serving, deployment, and infrastructure into a single platform.

Layer 6 — Managed API layer
Purpose: Consume optimized inference without owning the stack
Solves: Infrastructure ownership

Most companies don’t want to own the inference stack, but they also don’t want the limitations of a one-size-fits-all API.

Running models efficiently means making decisions about serving engines, deployments, routing, GPU provisioning, batching, autoscaling, and infrastructure. Companies like Fireworks, Together AI, and Doubleword take ownership of those decisions so developers don’t have to. You still interact with a familiar OpenAI-compatible API, but underneath the platform is constantly optimizing where your model runs, how requests are scheduled, how aggressively they’re batched, and how the underlying infrastructure is tuned for latency, throughput, and cost.

Fireworks AI
Built from the infrastructure up. Owns the serving layer, GPU infrastructure, and deployment stack to deliver highly optimized managed inference.

Together AI
Started as the cloud for open-source models and has expanded into a full-stack inference platform spanning training, fine-tuning, deployment, and inference.

Doubleword
Takes a different approach. Instead of asking developers to think about infrastructure, it asks them to describe the workload- realtime, async, or batch- and figures out the most efficient way to execute it underneath.

The interesting part is that they’re all converging toward the same developer experience. Fireworks and Together started by abstracting infrastructure. Doubleword started by abstracting workload orchestration. Different implementations, increasingly similar abstraction.

Market signal: The bottom of the stack is fragmenting into specialized layers. The top of the stack is converging back into managed platforms that bundle those layers together behind a familiar API.

Layer 5 —Agent-Native Inference
Purpose: Optimize long-running agent workloads
Solves: Long-running execution

This is probably the newest layer in the inference stack.

Everything we’ve talked about so far assumes inference is the thing being optimized. But long-running agents expose another bottleneck. They don’t just make model calls—they execute code, call tools, wait for inference, resume execution, and repeat that loop for hours. Optimizing inference alone leaves a lot of efficiency on the table.

That’s why I think we’re starting to see a new category emerge.

Sail Research

I first came across Sail Research when they were positioning themselves as another inference provider. The pitch was familiar: cheaper inference, better utilization, infrastructure built for AI agents.

Their recent product launches made me realize they’re becoming something different.

Instead of optimizing inference in isolation, Sail couples inference and execution together. Sailboxes are persistent execution sandbox environments where agents actually run. When an agent blocks on inference, the sandbox automatically pauses and resumes when the response arrives. You’re no longer paying for allocated compute while the agent is simply waiting.

They’ve since added Voyages, an observability layer for monitoring long-running agent workflows. Together, inference, execution, and telemetry become one system.

I think that’s what makes this layer interesting. It’s no longer just about serving models faster. It’s about optimizing the entire lifecycle of an agent.

Market signal: Agent runtimes are still an emerging category, but they’re one of the first infrastructure layers built specifically for long-running AI agents rather than conversational AI

That’s my attempt at mapping where inference is headed.

It took me way longer than I expected because the market is evolving incredibly fast. New companies seem to pop up every week, the boundaries between categories are already starting to blur, and everyone is trying to solve a slightly different piece of the same problem.

The interesting part isn’t any one company.

It’s that inference has quietly become its own software industry.

I think we’re still very early.

Further reading & acknowledgements: I'd highly recommend reading Inference Engineering by Philip Kiely and the Baseten team; 's comparison of vLLM, Ollama, SGLang, and TensorRT-LLM on The AI Engineer; ’s and the team at Wing VC, for some of the most thoughtful writing on inference I've read. Thanks also to my friend , founder of Muna, for the many conversations that helped shape this framework, especially around how the different layers fit together.

That works today because the number of decisions is still manageable. But as agents become more capable, asking a human to validate every tool call, every API request, and every execution-time decision quickly stops being practical. At some point, the human becomes the bottleneck, and you’ve lost the benefit of having an autonomous system in the first place.

That’s where I think the real governance problem starts.

Gartner recently predicted that by 2027, 40% of enterprises will demote or decommission autonomous AI agents because governance gaps won’t become apparent until after production incidents. I don’t think we have to wait for full autonomy to run into those problems.

If you’ve used Claude Code, you’ve already experienced the problem

You just probably haven’t thought about it as an AI agent governance problem.

You give Claude Code a task, and a minute later it’s read half your codebase, called a handful of tools, maybe spun up a sub-agent, edited a file, hit an API, and somehow found a config file you forgot existed. Most of that happened before you had a chance to follow along.

That’s not because Claude Code is doing anything wrong. It’s because that’s how capable AI agents work. They gather context, adapt their plan, choose which tools to call, and make decisions based on what they learn while they’re executing.

What struck me is that almost all of those decisions happened after I gave the agent the task.

That’s the governance problem.

Today, most AI agent governance is front-loaded

Most teams still focus on governing what an AI agent starts with, even though the most important decisions happen after execution begins. That model works well for deterministic software because the execution path is largely known in advance. AI agents are different. They gather new context, adapt their plan, and continuously make decisions while they're running.

The natural instinct is to apply least privilege, just as we do for humans. Give an agent the minimum access it needs. Scope the tools it can use. Issue short-lived credentials. That’s still good security, and it isn’t going away.

But least privilege isn’t the whole story anymore. It tells you what an agent could access, not whether a specific action should happen in the context of the task it’s executing.

An agent might have permission to GitHub, Jira, and Salesforce. The harder question is whether it should update this issue, modify this repository, or read this customer record while acting on behalf of this user.

That’s a runtime decision.

Right now, most teams think about AI agent governance in three layers.

Tier 1: Static governance. Acceptable use policies. Approved model lists. Compliance documents legal signed off on. These define what agents are supposed to do. They say nothing about what agents actually do.

Tier 2: Structural governance. Tool allowlists. MCP connectors. Sandboxed APIs. Better than documentation because you’ve narrowed the blast radius, but it’s still a configuration decision made before the agent starts.

Tier 3: Instructional governance. CLAUDE.md. AGENTS.md. .cursor/rules. System prompts. The agent harness. “Never access production.” “Always ask before deleting.” This is where most sophisticated teams are today.

The problem is that all three layers stop before execution.

Once the agent starts reasoning, choosing tools, and interacting with systems, the first three layers have already done their job. They shape the agent’s behavior, but they don’t enforce it.

That’s why I think there’s a fourth layer.

Tier 4: Runtime governance. Instead of evaluating policy once at setup, policy is evaluated at the moment of every tool call. Before the agent touches an external system, the runtime has the context of the task, the identity of the user the agent is acting for, the requested action, and the applicable policies. It can make a deterministic decision: allow or deny.

That's the point where governance becomes enforcement instead of guidance.

Before the agent touches anything

What runtime governance of AI agents actually looks like

The easiest way I’ve found to think about it is to start with how we already govern people, then ask what changes when the actor is an AI agent.

The same principles still apply: identity, authorization, and access. The difference is that those decisions now have to account for who delegated the task, what the agent is doing, and what it’s trying to access at that moment.

Identity - Who is this agent?

Agents don’t look like users. They spin up, perform a task, call a few tools, and disappear. Runtime governance starts by giving every agent a real identity that’s tied to who delegated the task, what it’s doing, and what scope it was given. Without that, everything that follows becomes difficult to attribute.

Runtime authorization - Should this action be allowed?

The second capability is runtime authorization.

Every tool call becomes a policy decision. Before an agent touches an external system, something has to decide whether that specific action should be allowed for this user, through this agent, on this resource, in the context of the task it’s currently executing.

What’s interesting is that the industry seems to have independently converged on where that decision belongs. Claude Code, Codex, Gemini CLI, and Microsoft’s Agent Governance Toolkit all expose hooks that let developers inspect or intervene while an agent is running.

The enforcement point already exists.

The challenge is making those decisions with enough context to determine whether the action is actually authorized.

Access - How does the agent get the permissions it needs?

The final capability is access.

Long-lived API keys, service accounts, and .env files don’t fit well with autonomous agents. Instead, access should be granted just in time, scoped to the specific action being performed, and expire as soon as that work is complete.

Taken together, that’s what runtime governance looks like: identity that can attribute every action, authorization that evaluates every tool call in context, and access that’s granted only for the work the agent is performing.

The Emerging AI Agent Governance Stack

One thing that stood out to me while researching this space is that nobody is building AI agent governance as a single product.

Instead, the market is emerging one layer at a time. Companies are building different parts of what increasingly looks like an AI agent runtime governance stack: agent identity, runtime policy evaluation, access & credentials, and audit. Once you look at the landscape through that lens, it becomes much easier to understand.

What’s also interesting is that the market is splitting into two camps. Some companies are building agent-native infrastructure from scratch. Others are extending the identity and security infrastructure enterprises already rely on.

Agent Identity

Identity is where I think the market gets interesting. There seem to be two approaches emerging.

The incumbents already own human identity. Okta, Microsoft Entra, Auth0, Ping Identity, and others aren't going away. If enterprises are going to deploy millions of AI agents, it makes sense that those platforms will eventually need to understand agents alongside people. The question is whether agent identity becomes another object inside existing IAM systems, or whether it requires an entirely new identity layer.

Keycard is the clearest example. Rather than treating an agent as another service account, it gives agents first-class identities tied to the human who delegated the task, the workload they’re executing, and the scope they were granted. It federates with existing IAM systems instead of replacing them, then adds delegation, workload attestation, and short-lived credentials on top.

There’s also an adjacent category that’s becoming increasingly important: non-human identity (NHI) governance.

Companies like Oasis Security aren’t creating identities for new AI agents. They’re helping enterprises discover and govern the machine identities they already have-service accounts, OAuth applications, API keys, and other credentials that autonomous agents increasingly inherit. As AI agents become another consumer of machine identities, NHI governance starts looking like a foundational piece of the runtime stack.

Runtime Policy

Once an agent has an identity, every tool call becomes a policy decision.

This is where companies like Cerbos and Oso fit.

Cerbos approaches the problem as an authorization policy engine. Rather than embedding permission logic throughout application code, it evaluates every authorization request against declarative policies at runtime. For AI agents, that means every tool call can be evaluated independently before it's allowed to execute.

Oso approaches the problem from the application authorization side. It helps developers define fine-grained authorization based on roles, relationships, and context, making it possible to evaluate what an AI agent should be allowed to do on behalf of a user instead of simply inheriting all of that user's permissions.

Microsoft’s open-source Agent Governance Toolkit is another interesting signal. Rather than inventing a new runtime, Microsoft is extending existing agent frameworks with runtime policy enforcement. To me, that’s a sign they see runtime policy evaluation becoming foundational infrastructure rather than a product differentiator.

Access & Credentials

Identity answers who.

Policy answers whether.

Something still has to answer how an AI agent obtains access.

That’s where companies like Arcade and Nango fit, although they’re solving different parts of the problem.

Arcade focuses on delegated, just-in-time access for AI agents. Instead of giving an agent a long-lived API key or service account, it obtains a short-lived credential on behalf of the user, scopes it to the specific action, and expires it when the work is complete.

Nango approaches the problem from the OAuth and API integration layer. It provides infrastructure for OAuth flows, token refresh, and credential management across hundreds of third-party APIs while allowing organizations to retain control of their own credential infrastructure.

Both are moving away from standing credentials toward credentials that exist only for the duration of the work an AI agent is performing.

Audit

Audit is less of a standalone product category than an outcome of the other three working together.

If identity, policy, and credentials are all evaluated at runtime, an audit trail naturally falls out of the system. You know who delegated the task, which policy evaluated the request, what credential was issued, and every action the agent took.

Companies like Keycard, Cerbos, and Arcade all contribute different parts of that picture - identity, policy decisions, and execution events, but together they make it possible to reconstruct what happened after the fact.

That’s the interesting part to me.

These companies aren’t building competing products. They’re building different layers of what looks increasingly like the runtime stack for AI agents.

Executive Order 14409 Signals Where AI Agent Governance Is Headed

On June 2, 2026, President Trump signed Executive Order 14409. Most of the attention went to Section 3, which targets frontier model developers.

I think Section 4 is the part enterprises should be paying attention to.

It directs the Attorney General to prioritize enforcement of existing laws, including the Computer Fraud and Abuse Act (CFAA), against people who use AI to gain unauthorized access to computer systems. It explicitly calls out AI agents.

The Executive Order doesn’t create a new legal standard for enterprises.

What it does signal is that authorization is becoming the question.

Not:

Did we tell the agent what to do?

But:

Can we prove this action was authorized?

To me, that's where AI agent governance is headed. Not more prompts or policy documents, but runtime identity, authorization, access control, and auditability that can answer that question every time an AI agent takes an action.