Better models make continuous integration (CI) a bigger share of the agent loop

‍^{Disclosure: Namespace is a Scale portfolio company.}

A coding agent works in a loop: it writes a change, checks it, and tries again until the checks pass. The checks come in two tiers. 

The cheap tier runs on the agent's own machine and covers linters, the unit tests relevant to its changeset, and a targeted subset of the end-to-end tests. The agent works out which tests are relevant, so it never runs the whole suite locally. 

The expensive tier is full CI, which runs only on pull requests into main. 

Each new model release shortens the time the agent spends writing code, but not full CI runtime. To the contrary, the models are great at writing more and more tests and code that slow things down. For Deckhand, our background coding agent and personal agent framework, the full CI run on a pull request into main had reached a median of 36 minutes.

Two kinds of work, two paths to main

The first kind are small, isolated jobs: patching a CVE, triaging a Sentry error, building a small feature. The agent finishes these in under ten minutes. It runs the cheap checks, opens a pull request into main, and waits for full CI. Nothing is shared between changes, so each one pays for its own run. The work took under ten minutes. The run behind it took 36.

The second group are large features, and they take a different path. We typically open one pull request per epic. The agent builds the epic as many small, short-lived branches that merge into a release branch, with only the cheap checks on the way in. The release branch then goes to main in a single pull request, and that is where full CI runs. Full CI runs once for an epic's worth of changes, and a reviewer sees one coherent unit of work.

Slow CI used to be invisible

Slow CI is tolerable for as long as people do the waiting, because we’re slow either way.

A person opens a pull request and starts something else, so a slow runner costs nothing anyone can see. An individual used to do one or two PRs a day by hand, so the CI load simply didn’t add up to much.

Namespace's Hugo Santos made a version of this argument in a recent talk: human writing and review time hid the latency of slow machines, and cheap code generation removes that cover. The release-branch path is close to the shape he describes, with agents producing many small pieces cheaply and the reconciliation happening once, before main.

Why we keep the checks

This year, our merged pull requests to main have grown from about 100 a month to more than 600, with fewer developers. Each of those PRs now runs more than it used to: linting, unit tests, a wider test suite, end-to-end tests and security scanning. Every PR pays for all of them, so the total grows with the number of checks and with the number of PRs. 

Cutting checks would shorten the wait, but each check is a piece of manual code review a person no longer has to do. 

The path here is clearly not to do less testing, linting, scanning; we need to do more to ensure our code is actually of requisite quality. We need to build our systems to be both performant and efficient: running each check quickly and only at times where the check matters. Because the shape of work done by agents is different than the shape of work we used to do as humans, the calculus on that has changed.

The runner comes first

One common response is to redesign CI for agents: better cache keys, test selection, or verification that runs inside the agent's loop before a pull request exists. We already do some of that, because the agent chooses which tests to run locally. At our size we think the next step is the cheaper one: check what the pipeline runs on.

Ours ran on GitHub's default runner for a private repo, which has 2 vCPUs. We had spent October working around it, with releases taking hours. Eventually, fed up,  we changed the runner label on all seven jobs from ubuntu-latest to nscloud-ubuntu-24.04-amd64-4x8, a Namespace runner with 4 vCPUs and 8 GB of memory. 

Namespace was the obvious place to look. Other than the fact that we are proud investors, Deckhand's agents themselves already run on Namespace! 

The workflow is still GitHub Actions YAML, and the change to each job was the runner label. 

To make sure we were using our new resources fully, we also adopted pytest-xdist to add parallelism across our new cores, and split our test suite into 2 tiers: fast and slow. The fast tier runs on every PR (including those short lived temporary branches) and includes a timing check where it fails no matter what if it gets longer than a couple of minutes. The slow tier runs only on merges into main.

What it got us

The median full workflow on all-green runs fell from 36 minutes to 8.9. The test job fell from 36.5 minutes to 8.7, but that figure mixes the hardware with the parallelism and the removed duplicate run. The cleaner number is the fast tier, which we did not touch: it went from 276 seconds to 92, about 3x from the machine alone, and it now runs well inside its original 240-second budget.

Agents have changed what CI has to do. They write code faster than any person did and send far more of it through the pipeline, so the minutes a change spends waiting add up in a way they never did when one engineer opened a pull request or two a day. Getting the full run from 36 minutes to 8.9 took a different runner and some parallelism, and it was the cheapest improvement available to us. 

The next gains will come from the structure around the runner: which checks run on the agent’s machine, which run on every branch, and which wait for main. 

‍