How we use AI to build a better map of the venture landscape

The Problem

Max Abram, a Partner at Scale, started building conviction in late 2025 around a specific pattern: AI-native companies using software and agents to do what systems integrators have traditionally done for enterprise deployments, including ERP migrations, SAP customization, Salesforce implementations, Workday rollouts, or legacy code modernization. His belief is that there’s a bright future for companies that compress timelines requiring large teams of expensive consultants into something much faster and cheaper.

In an attempt to map this sector, he manually assembled a list of 10 or so companies he'd sourced, met with, or tracked over time. He suspected he was missing others, but he ran into a problem: This category was so new that nobody could agree on a name for it. Individual startups had their own names for the concept, using terms like “AI-driven DevOps automation” or “no-code workflow platforms.”

It made it difficult for Max to use existing vendor databases and search engines to find the kinds of opportunities he was looking for – let alone finding fair comps. And it wasn’t just Max, either: Every investor runs into this problem in one form or another, as they rely on a company’s own language to parse what they actually do or what their business model is like.

That’s why we built a continuously-updated semantic map of the venture landscape: every company in our dataset embedded in a shared vector space and clustered by structural similarity using our decades of in-house data, not just terminology.

When we ran Max's AI implementation market through this system, we found 35 companies sharing the same structural pattern across what the taxonomy had scattered into five different categories. The system didn't figure this out via pre-existing labels or categorization — it found them because they were actually similar. Six of those companies hadn’t even been on Max’s radar before.

What We Built

The pipeline behind this engine works in five stages.

First, we ingest company data from multiple sources on a recurring basis: market data vendors, public filings, news feeds, web scraping, and our own historical deal flow. Web scraping in particular fills gaps that structured vendors miss, company websites, job boards, and founder profiles often carry signals that haven't made it into any database yet.

Second, each company gets processed by a language model that extracts structured signals: business model, industry, sub-industry, and the specific technologies it uses or produces. We made an early decision to use constrained, structured outputs here rather than just free-text descriptions. That decision compounds: structured labels are easy to compare across tens of thousands of companies; free text isn't.

Third, we embed each company's description and taxonomy into a 1024-dimensional vector space using Cohere's Embed v4, stored in PostgreSQL with pgvector. Companies doing structurally similar things end up near each other in this space regardless of the words they use to describe themselves.

Fourth, we cluster over that vector space to produce groupings that reflect actual similarity. This is where the secret sauce is. Our decades of memos and analysis done to identify true competitors, comps, and nuances where two companies appear similar but aren’t feed directly into our custom semi-supervised clustering algorithms built from the ground up for this exact use case.

Finally, we pass samples from each cluster back through a language model to generate market names, descriptions, and boundary characterizations. The taxonomy names aren't imposed top-down from analyst priors. They emerge from what the companies in each cluster actually are.

After the initial clustering pass, we ran a Cowork agent over the category graph—pulling member companies from adjacent clusters, cross-referencing overlap, flagging companies that shared the structural pattern but hadn't appeared in normal deal flow.

That combination, a structured map plus an agent that can systematically traverse it, is what changes the economics of market research. Without the map, the agent is inundated with hundreds of thousands of companies and no learned institutional knowledge of how they relate to each other. Without the agent, the map is a static report you look at once. Together, they make it possible to ask a real question about the shape of a market and get a real answer in minutes rather than weeks.

How We Built It

The biggest technical decision was owning our own description and tagging models rather than relying on vendor descriptions and taxonomies. This let us build consistently formatted and structured data that actually maps into the vector space well, and lets us update the data automatically with the non-public information we learn in our daily work.

The second key decision was to build our own hierarchical temporal clustering framework. We have large markets made up of small categories. Categories split, merge, retire, and emerge. A static taxonomy goes stale. Our clusters update as companies evolve; a company that pivots shows up in a different cluster on the next run. But we also made the whole system for humans, so we needed lineage-aware models that only change things when they really need to, so our investors can do the real work without every tag changing every week.

It’s a lot of math and data infrastructure. We’re using billions of tokens a month, doing large scale scraping and data engineering work, and training custom models, but the result is substantial. On a held out test we’re able triple baseline performance in terms of must-link pairs being in the same resulting clusters vs more a pure vendor-data approach.

The distinction between technologies a company uses and technologies a company produces turned out to matter more than we expected. "Uses AI to do taxes for small businesses" and "uses tax data to train a tax-specific foundation model" are completely different markets with different metrics, competitors, and dynamics.

Lessons Learned

Coverage is a constraint you have to be honest about. The clustering reflects the companies we have data on. Geographies, sectors, or early-stage companies that haven't surfaced in any of our sources produce gaps in the map. We work to expand coverage, but it's an ongoing effort, not a solved problem.

The embedding model doesn't always capture the distinctions that matter for venture. Some differences that look large in vector space are commercially irrelevant; some that matter enormously are subtle enough that companies end up with similar embeddings. So descriptions and embeddings just doesn’t get you there, semi-supervised approaches with high quality labeled data are a must.

The cluster boundary cases are often the most interesting companies: the ones that straddle two emerging categories, or that sit in their own tight cluster before a market name exists for them. We've learned to pay more attention to the outliers than the clean cluster members.

The taxonomy is currently the foundation for most of our analytical work: market sizing, competitive landscape analysis, identifying companies doing structurally similar things to a company we're evaluating. The next step is making it more interactive: agentic research on the markets themselves, investor and founder mapping and tracking within them, and more.