Inference providers have their head in the clouds

On April 23, 2015, Amazon did something it had refused to do for 9 years: it told the world how much money AWS made. Up to that point, AWS had been buried in a line item called “other,” lumped in with its credit cards and advertising. Analysts could see “other” growing fast, but nobody knew whether the cloud business was a gold mine or a money pit. 

AWS was also facing relentless price competition from companies with deeper pockets, and the expectations were that margins would get ground to nothing. Renting servers was a commodity business, and commodity businesses go to whoever can afford to lose the most money for the longest. Wall Street modeled the business at negative margins, with some estimates having AWS losing 20 cents on every dollar.

Then the numbers came out. $1.5 billion of revenue in the quarter, up 50% from the last. Amazon’s market cap jumped almost 15% overnight. Ben Thompson called it the AWS IPO, and pointed out that the implied valuation of a business most people had written off was higher than Google’s at its actual IPO. Three months later Amazon reported a surprise profit, and the stock had its best day in five years. Today AWS does more than $46 billion of revenue a quarter at a ~40% operating margin. This “commodity” business turned out to be one of the best businesses ever built.

Neoclouds and inference providers today look a lot like AWS did early on: companies renting someone else’s hardware at thin margins, in a market almost everyone agrees is a commodity. I think there’s a real bull case for them, but it’s a narrow path. You have to keep cutting prices on the commodity layer to stay competitive, while using that business to keep shipping products on top that solve harder and harder problems for your customers, which is where the real value is. 

This path is narrower than it was for AWS. Climbing up the stack matters more, because the bottom layer is a worse business: even when everyone thought AWS was a commodity, it was still doing 17% operating margins, while CoreWeave did a 5% adjusted operating margin last quarter. But climbing is also harder, because there’s more competition today than in 2010. Most providers won’t manage to stay on the path, but the few that do could end up looking a lot more like AWS than anyone expects.

The AWS playbook

If you were a tech company in 2006, getting servers meant guessing your capacity months in advance, buying hardware, signing a colo contract (or finding an office in Santa Clara with a big enough server room), and waiting weeks for it all to be racked and networked. A huge share of your seed round went to just buying and setting all this up. AWS changed this by turning all this work into an API call. Making that work was genuinely hard engineering that almost nobody else could do.

Within a few years, however, every major tech company and a long tail of hosting providers could rent you a virtual server, and the raw compute that had been AWS’s breakthrough was on its way to becoming a commodity. If that had been the whole product, the analysts in 2015 would have been right. After its initial breakthrough, AWS relentlessly pursued two goals.

First, it cut the price of the “commodity” layer. The first EC2 server cost 10 cents an hour in 2006. The comparable instance today runs about 2 cents an hour, roughly 80% cheaper for a much better machine. Storage fell even further, with storing a gigabyte in S3 costing 15 cents a month at launch and about 2.3 cents a decade later. Every price cut made raw compute and storage a worse business for anyone trying to compete, and pulled more customers onto AWS.

Second, AWS kept pushing products and climbing. In 2009 it launched RDS. In 2012 it launched DynamoDB, then Redshift, then Aurora, then Lambda. Each one took a bigger, more painful problem off the customer’s plate. Data warehouses that used to take quarters to stand up took minutes, at a fraction of the cost.

All the new products are what kept customers around. Once your data warehouse is in Redshift, your app runs on RDS, you have a bunch of Lambda functions wired into everything, and years of data are sitting in S3, moving to save a few cents an hour on compute stops making any sense.

How an inference company climbs the stack

In 2023, if you had a relationship with Nvidia (or you knew someone who knew someone at Nvidia), then you could get allocation when nobody else could, and you could build a very lucrative business as a compute provider. You signed a long term contract for a few thousand H100s, sliced it up into smaller contracts, and resold the slices to customers who couldn’t get a cluster any other way. That was the whole business! Your customer could leave you tomorrow if someone else had capacity available, and the only thing keeping them around was that nobody else did. Gross margins reflect that at 10 to 15% on a good day. Revenue explodes, but you’re less of a tech company and more of a reseller.

One level up, you stop handing someone a raw cluster and start running models for them. You build a serving stack (batching, kernels, quantization, routing) and sell tokens through an API. In 2023, that solved a problem most customers genuinely couldn’t solve themselves. They didn’t have the ML engineers to optimize inference and didn’t want to hire them, so you were doing the dirty work to give them what they actually wanted, which is tokens. The problem is that by 2026 a lot of that work has been commoditized, partly because open source serving engines like vLLM / SGLang made it much easier. Every serious provider can serve the same open-weight model at roughly the same price, and switching means changing an API endpoint.

Once you’re serving tokens for a few hundred customers, you start seeing what they actually struggle with. They need the model fine-tuned on their data. They need a data pipeline to collect and clean that data in the first place. They want better capabilities in specific areas, so they want RL. They need the custom model served back to them with their integrations wired in. Each of those is more valuable than tokens, because it’s a harder problem the customer can’t easily solve alone, and each is a deeper entanglement with the customer’s workflow. If you’re running someone’s custom fine-tuned model with their data pipeline feeding into it, they can’t leave without finding an ML engineer who can take the model weights, migrate to another provider, and re-optimize everything. That’s a six month project nobody wants to do when things are working.

Once you’re that embedded with a customer, you have a platform you can sell other products on top of: sandboxes, dev tooling, observability, whatever the customer needs next. Modal has done this well with sandboxes as an upsell on its compute. These are the products that get your customers to stick around and fall in love with you.

A surprising number of the companies near the top today started at the bottom, and I don’t think that’s a coincidence. If you’re running a customer’s inference, you can see exactly where they get stuck and that’s how you know what to build next. You already have the relationship, so the next product is an upsell rather than a new sale. You already have engineers who know how to run models fast and cheap on your hardware, which is most of what you need to fine-tune and serve them too. AWS was in the same position: it could build RDS and Redshift because it was already running its customers’ servers and could see what they were trying to do with them. Which is also why, in 2023, the bottom of the stack looked like a much worse place to start than it turned out to be.

What I would have gotten wrong in 2023

I didn’t start investing until 2025, so I don’t have the visceral pain of passing on one of these companies early and watching it go up 100x. But I was following them in 2023 and 2024, and I’m pretty sure I would have passed when I shouldn’t have. Inference providers were reselling someone else’s chips at 10 to 15% margins and selling the same thing everyone else was selling. If you’re just slicing up compute contracts and reselling them at a markup, there isn’t a lot of enterprise value being created. I still think that’s true, but looking back that view got two things wrong.

The first is the same mistake the analysts made with AWS in 2015: they looked at the servers, correctly priced them as a commodity, and missed that the servers were how AWS got customers to sell everything else to. The teams that started out just getting people GPUs were the ones in the best position to climb up the stack, and they are the ones that are now selling fine-tuning, RL, custom model serving, and developer tools to customers they first won with capacity. The low-margin business turned out to be how they earned the right to build the better one.

The second is that it assumed the hyperscalers would take this market anyway. AWS, Azure and Google Cloud had the capital, the customers, and the software, and it seemed obvious they’d absorb AI compute the way they’d absorbed everything else. What that missed was that Nvidia didn’t want them to. The hyperscalers buy a huge share of Nvidia’s chips and are also building their own to replace some of that spend, so Nvidia had every reason to build up other places to sell GPUs. The neoclouds effectively became Nvidia’s cloud. The same dependence on Nvidia that makes the path narrow today is a big part of why there was a path in the first place: without it, the hyperscalers would likely have locked up the customers early on.

So the question I should have been asking in 2023 wasn’t “is this a good business?” It obviously wasn’t yet. It was, “Can this team run both halves of the AWS playbook, staying competitive on the commodity layer while building the products on top?” That’s a lot harder to underwrite than a margin, which is probably why so few people did it, and why I don’t think I would have either.

What I’d look for now

My sense is that it’s harder now to start at the bottom and climb than it was in 2023. Back then the market was early, customers needed someone to figure things out for them, and there was room to grow up with the market. Today the providers that already climbed have scale and customers, and they’re building into the same layers a new company would want to. The hyperscalers are moving aggressively, too, with Bedrock being the fastest-growing multi-billion dollar service in AWS’s history. Starting from raw capacity today means scaling compute and building software at the same time, against companies that are already a few rungs up. 

The companies that seem best positioned to me are the ones where the software is already part of what customers pay for, whether that’s fine-tuning, serving custom models, or RL infrastructure, and where leaving would mean real work for the customer. That kind of start feels like it can grow into a big business, because each new product builds on a relationship the customer already depends on.

Modal started with developer infrastructure, Thinking Machines put out Tinker as a fine-tuning and RL product, and Applied Compute is building and serving custom models for companies. These teams already have the second half of the playbook: they’re selling the products that are valuable and sticky. Their challenge is the first half, getting compute cheap enough that someone with their own GPUs can’t undercut them. But it seems to me that’s an easier problem to solve with customers already in hand than the other way around.

Which brings it back to AWS. Renting servers was never a great business on its own, even for AWS; it was how AWS got customers in the door to sell them everything else. I think most GPU clouds and inference providers won’t manage to run both halves of that playbook, and they’ll stay commodities no matter how fast they grow. But the few that do, staying cheap on the compute while solving bigger and bigger problems for their customers on top of it, could turn into much better businesses than most people expect, the same way AWS did.