So far, most companies building AI products have been in what I call the accept and consume wave.
The playbook was simple: pick a frontier model, call the API, and build your product around whatever latency, pricing, throughput, and rate limits the provider offered. That wasn’t because companies didn’t care about inference. It was because they were trying to build useful products as quickly as possible. Foundation model APIs made that incredibly easy. They handled the infrastructure, gave developers instant access to frontier models, and continuously upgraded those models over time. If paying a little more for inference meant shipping months faster, it was an easy trade.
That tradeoff is starting to break.
We’re entering the second wave: own and optimize.
As AI products mature, inference is no longer something you simply consume - it’s becoming something you optimize. Not just because it’s expensive, but because inference is becoming an application-specific systems problem. A voice AI company, a coding copilot, a research agent, and a document processing pipeline are all calling an LLM, but they have completely different latency, throughput, concurrency, and cost requirements. The default API has to optimize for everyone. Your application doesn’t.
That naturally raises the question: why now?
What changed over the last two or three years that suddenly made inference something companies want to own instead of simply consume?
It wasn’t one breakthrough. It was a handful of shifts that all happened at roughly the same time.
What Changed?
The forces reshaping how companies consume inference
01 — The Workload Changed
From Conversational to Agentic
Most AI applications up until this point were conversational or workflow-based. A user would submit a prompt, maybe the application would make a couple of LLM calls behind the scenes, and then return an answer. Teams spent most of their time improving prompts, adding examples, tweaking outputs, or chaining together a handful of predefined steps. Token usage was relatively predictable because the application controlled the workflow.
Agentic systems are different.
A single user prompt is no longer a single request. It’s the beginning of an execution loop. The model reasons, plans, decides whether it needs to use tools, retrieves context, calls those tools, evaluates the results, and repeats until the task is complete. If you’ve used Claude Code or Codex, you’ve already seen this. You ask it to fix a bug, and behind the scenes it searches your codebase, opens files, writes code, runs tests, retries when something fails, and keeps going until it has a solution.
Figure 1. A chat interaction usually ends after one or two model calls. An agent loops through planning, tool use, evaluation, and retries until the task is complete.
The important shift is simple: inference no longer scales with prompts. It scales with the amount of work an agent performs on your behalf. Every reasoning step, tool call, retry, or planning step can trigger another inference request. Some tasks might finish in a handful of model calls. Others can require hundreds.
Figure 2. The economics change quickly. What was pennies for a chat session becomes meaningful infrastructure spend once agents continuously execute work.
The surprising part is where those tokens actually go.
Most people assume they’re paying for the final answer. In reality, a large portion of the inference bill is everything required to get there: conversation history being carried forward, system prompts, tool definitions, planning, retries, and intermediate reasoning. As workloads become more agentic, the overhead around the answer grows alongside the answer itself.
Figure 3. In an agentic workflow, much of the inference budget is spent on orchestration rather than the final user-visible output.
02 — The Constraints Changed
The workload changed. The optimization target changed with it.
Once companies started building agents instead of simple chatbots, they realized there isn’t one definition of “good” inference anymore. Every application is trying to optimize for a different combination of latency, throughput, and cost, which means there isn’t a single inference stack that’s optimal for everyone.
Underneath, every inference request has two phases. The first is prefill-processing the input prompt before the model can start generating a response. This determines Time to First Token (TTFT), or how long the user waits before seeing the first word. The second is decode-generating the response one token at a time. This determines Time Per Output Token (TPOT), or how quickly the rest of the response streams back. Most optimizations in the serving stack are trying to improve one or find a better balance between the two, without blowing up cost.
Figure 4. Different AI applications optimize for different combinations of latency, throughput, and cost. There isn’t a universally optimal inference stack, only one that’s optimal for the workload.
This is where the tradeoffs become obvious. A voice AI company wants the model speaking in a couple hundred milliseconds because every delay feels awkward. A coding copilot also cares deeply about latency, but it has to maintain that experience across thousands of concurrent users. A research agent running overnight is happy to wait a few extra minutes if it cuts inference costs in half. A batch document processing pipeline doesn’t care about latency at all, it wants maximum throughput and GPU utilization.
They’re all building agentic applications. They’re all running inference. But they’re solving completely different optimization problems.
That’s why the default API stops being enough. Foundation model providers have to optimize for millions of developers across millions of different workloads. Your application only has one workload. The closer inference becomes to your product, the more those default tradeoffs stop making sense.
03 — Open source caught the frontier
DeepSeek, Qwen, Llama, Kimi K2, GLM- the open ecosystem has matured quickly. There isn’t one “best” model anymore. Different models lead on different benchmarks and different workloads. Some are better at reasoning, others at coding, multilingual tasks, or long-context retrieval. That gives companies something they didn’t really have two years ago: choice.
Where is inference actually being optimized?
By this point, it’s clear inference is no longer just an API call. It’s an optimization problem. So where is all of this innovation actually happening?
The answer is across the entire stack. Some companies are rebuilding hardware. Others are rebuilding the software layer sitting on top of it. Both matter, but they operate on very different time horizons.
Hardware (Brief, But Honest)
Inference isn’t just changing software. It’s changing hardware too.
Training and inference stress hardware differently. Training is dominated by massive parallel computation. Inference is dominated by latency, memory bandwidth, and serving requests efficiently. That’s why we’re seeing a new generation of inference-first silicon alongside GPUs.
Three themes stand out.
First, specialized inference chips are becoming real. Companies like Groq, Cerebras, and d-Matrix proved there was room for architectures optimized specifically for inference rather than general-purpose GPU workloads. Whether those companies remain independent matters less than the fact that the industry has validated the need for inference-first hardware.
Second, hardware and models are starting to be designed together. Google has been doing this with TPUs for years, and OpenAI's reported Jalapeño effort points in the same direction. General-purpose GPUs have to support every workload. Custom silicon only has to optimize for one. The competitive advantage is no longer just the chip or the model, it's how well the entire stack works together.
Third, inference is pushing hardware toward full-stack systems. NVIDIA’s answer is Rubin. It isn’t just a new GPU, it’s an entire rack designed as one system, combining CPUs, GPUs, networking, interconnects, and software. That’s because inference isn’t just a compute problem anymore. Models like DeepSeek and Qwen3 constantly move tokens between experts running on different GPUs. The chip still matters, but so does everything connecting those chips together. The competitive advantage is no longer the accelerator. It’s the system around it.
Purpose-built inference hardware will absolutely matter.
But hardware refresh cycles happen over years. Software improvements ship every week.
That’s why the rest of this article focuses on the software stack. Today, it’s where companies are finding the biggest improvements in cost, latency, and throughput.
Software Stack
Inference isn’t one optimization problem anymore. It’s a stack of them.
Every layer exists because it’s solving a different bottleneck. Some make models portable. Others squeeze more work out of GPUs. Others manage deployments or optimize agent runtimes. Together, they determine the latency, throughput, and cost of every inference request.
Here’s how I think about the software stack, starting closest to the hardware and moving upward toward the application.
Layer 1 — Model Runtime
Purpose: Portable execution
Solves: Deployment fragmentation
This layer sits closest to the hardware. One of the messiest parts of AI today is that every piece of hardware wants your model in a different format. NVIDIA has one optimization stack. Apple has another. TPUs have another. CPUs have another. The model might stay the same, but the deployment pipeline changes every time.
Muna
Compiles Python inference code into optimized native executables, removing much of the runtime overhead and deployment complexity. Instead of shipping Python, containers, and a long chain of dependencies, you deploy a lightweight binary that starts faster and runs efficiently across different hardware.
Modular
Takes a broader approach by rebuilding the runtime itself. Think of MAX as the engine that runs AI models and Mojo as the language developers use to build AI software. Together, they’re trying to hide hardware-specific complexity so the same AI software can execute efficiently across CPUs, GPUs, and other accelerators.
Layer 2 — Model Serving Engine
Purpose: Maximize GPU utilization
Solves: GPU underutilization
This is where inference economics are won or lost.
Running a model efficiently is much harder than just loading it onto a GPU. Requests arrive at different times, prompts have different lengths, outputs finish at different times, and GPUs end up spending more time waiting than doing useful work. The job of the serving layer is to keep that hardware as busy as possible.
That’s where techniques like continuous batching, disaggregated prefill, and custom kernels (e.g. paged attention) come in.
Instead of serving one request at a time, they continuously pack new requests onto the GPU, reuse memory more efficiently, and dramatically improve utilization. The model doesn’t change. The hardware doesn’t change. You simply get more tokens out of the same GPU, which means lower cost and higher throughput.
Inferact
vLLM has become the default serving engine for production LLM deployments. If you’re serving chatbots, copilots, or general-purpose AI applications at scale, chances are you’re either using vLLM or evaluating it. Inferact is building the commercial layer around that ecosystem.
RadixArk
SGLang started with efficient structured generation but has quickly become a favorite for agentic and RL workloads, where models are reasoning, using tools, and running complex execution loops. RadixArk is commercializing that next generation of serving infrastructure.
Both companies were founded by many of the researchers behind the underlying open-source projects. It’s a familiar playbook: build the infrastructure in the open, then build the company around operating it at production scale.
Market signal: Both companies are betting that the serving layer becomes valuable in the same way Red Hat built a business around Linux - by commercializing critical open infrastructure rather than owning it.
Layer 3 — Deployment Platform
Purpose: Run inference reliably in production
Solves: Production complexity.
Getting a model to run efficiently is only half the problem. You still have to deploy it, scale it, monitor it, and keep it running as traffic changes. That’s the job of the deployment layer. These companies sit one layer above the serving engine, handling GPU provisioning, autoscaling, deployments, routing, observability, and everything else needed to operate inference in production.
Baseten
Built for production inference. Models stay warm, serve traffic continuously, and come with the operational tooling you’d expect from a production service. If your model is powering a customer-facing application with real latency and uptime requirements, this is the type of platform you build on.
Modal
Built for on-demand compute. Instead of keeping GPUs running 24/7, infrastructure spins up when your code runs and disappears when it’s done. It’s a much better fit for batch jobs, fine-tuning, parallel workloads, and agents that don’t need persistent endpoints.
The interesting part is that even though both abstract away GPU infrastructure, they just optimize for completely different workload patterns. Baseten assumes your model is always serving users. Modal assumes compute should exist only when there’s work to do.
Market signal: The line between deployment platforms and managed inference clouds is already starting to blur. Companies like Baseten, Fireworks, and Together are expanding vertically across the stack, increasingly bundling serving, deployment, and infrastructure into a single platform.
Layer 6 — Managed API layer
Purpose: Consume optimized inference without owning the stack
Solves: Infrastructure ownership
Most companies don’t want to own the inference stack, but they also don’t want the limitations of a one-size-fits-all API.
Running models efficiently means making decisions about serving engines, deployments, routing, GPU provisioning, batching, autoscaling, and infrastructure. Companies like Fireworks, Together AI, and Doubleword take ownership of those decisions so developers don’t have to. You still interact with a familiar OpenAI-compatible API, but underneath the platform is constantly optimizing where your model runs, how requests are scheduled, how aggressively they’re batched, and how the underlying infrastructure is tuned for latency, throughput, and cost.
Fireworks AI
Built from the infrastructure up. Owns the serving layer, GPU infrastructure, and deployment stack to deliver highly optimized managed inference.
Together AI
Started as the cloud for open-source models and has expanded into a full-stack inference platform spanning training, fine-tuning, deployment, and inference.
Doubleword
Takes a different approach. Instead of asking developers to think about infrastructure, it asks them to describe the workload- realtime, async, or batch- and figures out the most efficient way to execute it underneath.
The interesting part is that they’re all converging toward the same developer experience. Fireworks and Together started by abstracting infrastructure. Doubleword started by abstracting workload orchestration. Different implementations, increasingly similar abstraction.
Market signal: The bottom of the stack is fragmenting into specialized layers. The top of the stack is converging back into managed platforms that bundle those layers together behind a familiar API.
Layer 5 —Agent-Native Inference
Purpose: Optimize long-running agent workloads
Solves: Long-running execution
This is probably the newest layer in the inference stack.
Everything we’ve talked about so far assumes inference is the thing being optimized. But long-running agents expose another bottleneck. They don’t just make model calls—they execute code, call tools, wait for inference, resume execution, and repeat that loop for hours. Optimizing inference alone leaves a lot of efficiency on the table.
That’s why I think we’re starting to see a new category emerge.
I first came across Sail Research when they were positioning themselves as another inference provider. The pitch was familiar: cheaper inference, better utilization, infrastructure built for AI agents.
Their recent product launches made me realize they’re becoming something different.
Instead of optimizing inference in isolation, Sail couples inference and execution together. Sailboxes are persistent execution sandbox environments where agents actually run. When an agent blocks on inference, the sandbox automatically pauses and resumes when the response arrives. You’re no longer paying for allocated compute while the agent is simply waiting.
They’ve since added Voyages, an observability layer for monitoring long-running agent workflows. Together, inference, execution, and telemetry become one system.
I think that’s what makes this layer interesting. It’s no longer just about serving models faster. It’s about optimizing the entire lifecycle of an agent.
Market signal: Agent runtimes are still an emerging category, but they’re one of the first infrastructure layers built specifically for long-running AI agents rather than conversational AI
That’s my attempt at mapping where inference is headed.
It took me way longer than I expected because the market is evolving incredibly fast. New companies seem to pop up every week, the boundaries between categories are already starting to blur, and everyone is trying to solve a slightly different piece of the same problem.
The interesting part isn’t any one company.
It’s that inference has quietly become its own software industry.
I think we’re still very early.
Further reading & acknowledgements: I'd highly recommend reading Inference Engineering by Philip Kiely and the Baseten team; Paolo Perrone 's comparison of vLLM, Ollama, SGLang, and TensorRT-LLM on The AI Engineer; Chris Zeoli’s and the team at Wing VC, for some of the most thoughtful writing on inference I've read. Thanks also to my friend Yusuf Olokoba, founder of Muna, for the many conversations that helped shape this framework, especially around how the different layers fit together.
sharp map. one layer sits above all six though: the decision to run inference at all. caching, deterministic fallbacks, and cheap validation cut calls before they touch the serving stack. in an agent loop, the biggest win is the model call you avoid. thanks for the mention.
Thank you for sharing your thoughts and the map. I’m currently following very similar assumptions. I currently do a lot of research in inference time scaling because I think the consumptions of tokens will explode when we switching from conversational AI to fully agentic systems. Thank you!