We built a background coding agent named Sloppy Joe

The Problem

Every internal tool team has a feedback backlog. Ours was no different. Our investment team uses a suite of internal tools daily (the MCP server, scoring dashboards, the Chrome extension, our voting app, and more). Before, issues and ideas came up and were either kept in some users head (worst case) or sent via email, slack, sms, mentioned in a meeting or some other informal communication. We had notion pages for tracking things but users couldn’t find them, forgot about them, or simply didn’t have time in the moment to stop what they were doing to go write a ticket.

Not because anyone didn't care, the feedback is simply the exhaust of their work, not the focus.

Often the asks are simple and easy to implement, but by the time they get communicated, prioritized, worked on, and released the turnaround time was slow.

The same problem showed up on the maintenance side. Every night, Sentry would catch new errors from our production services. Ox Security would surface new dependency vulnerabilities and IaC findings. All of it went into queues that were longer than the time we had to work through them. Triage alone, reading a stacktrace, tracing it to the right file, deciding whether it's worth fixing now, was taking more time than it should.

We wanted to close both loops automatically. Not fully autonomously (a human still reviews and merges everything), but automatically enough that by the time an engineer sits down in the morning, the feedback from yesterday already has a draft PR waiting.

What We Built

We built a pipeline that takes feedback from our internal tools and turns it into draft pull requests overnight, with no engineer involvement until review time.

The first leg of this is recording the feedback. Tools like Sentry and Ox do a great job at finding acute issues (failures, CVEs, etc), but we wanted to go further to include tech debt, architectural issues, and user feedback.

All of our AI user interfaces share the same MCP for access to our internal systems of record and tools. We added a simple “feedback” tool to the MCP that allows any of them to log a bug or feature request, which gets written to a table in psql. In the tool definition we’re able to prompt the user’s agent to get a full and rich context, which gives much more detailed and useful tickets then we would ever get from a normal person typing in slack.

We also have a shared library of skills for developers that contain our architectural and code standards, giving a great foundation for identifying gaps, i.e. tech debt.

The other half of the system runs on two Lambda functions. A poller fires at 2 AM UTC and spawns our different agents (feedback, security, tech debt).

The feedback agent queries the feedback table for unprocessed submissions, and starts an Anthropic Managed Agent session for each one. Each session gets a full task description: the feedback text, who submitted it, what tool they were using, plus a system prompt that describes the codebase layout, the quality checks that must pass, and the exact format for the PR. Three hours later, a checker Lambda polls the running sessions, extracts PR URLs from the output, and records completion status back to the database.

The agent does the full implementation loop: reads the relevant source files, writes the fix, runs linting and tests, bumps the version and changelog, commits to a branch, opens a draft PR, requests a reviewer, and then waits for CodeRabbit to run its automated review—addressing any actionable comments before converting the PR back to draft for human review.

We built the same architecture for security and maintenance. A nightly findings triage agent pulls the previous 24 hours of Sentry errors and Ox Security findings, groups them by repository, and opens (or appends to) a rolling draft PR per affected repo with fixes for anything it could resolve. A tech debt agent, run manually on demand, sweeps repos for linting issues, dead code, missing observability instrumentation, and Dockerfile hygiene. Both use the same dispatcher and the same agent framework as the feedback pipeline.

Often these PRs are complex and require multiple PRs across different repos with dependencies on each other. Because we have a rich set of skills and documentation, the agents are able to navigate this well.

The ROI

The most direct impact is on turnaround time. Feedback that used to sit in a queue for days now has a draft PR by morning. For the person who submitted it, the experience changes from "I filed a thing that might get picked up someday" to "I filed a thing and there's already a PR." That feedback loop—fast enough to feel real—changes how much people use the feedback mechanism and how specific they're willing to be when they use it.

On the maintenance side, the shift is in what engineers have to do. Sentry errors and Ox findings that previously required a human to triage, diagnose, and fix are now largely handled before anyone looks at them. The ones that get through to human review are genuinely ambiguous—architectural questions, false positives, things requiring context the agent couldn't reconstruct. The noise is gone.

The system currently covers our main MCP server, deal scoring pipeline, infrastructure code, and several supporting services. On a typical night it processes somewhere between a handful and a dozen feedback submissions, plus whatever Sentry and Ox surface. The sessions run for 30–60 minutes each and have a meaningful success rate on straightforward changes.

Lessons Learned

The fail-closed rule was the most important design decision we made, and we made it reactively rather than proactively. Before we added it, the system was technically functional but unreliable in a hard-to-see way: agents would encounter an environment issue, substitute a weaker check, open a PR with a green checklist, and the failure wouldn't surface until CI ran. Now any environment issue produces a clearly failed session rather than a quietly broken PR. The engineer who reviews the output knows exactly what went wrong.

The CodeRabbit integration added huge value. We’re heavy users for human and agent written code and have built up significant context, so allowing CodeRabbit to do the first pass of Sloppy Joe’s work meaningfully increases the rate of mergeable PRs for us on first read.

The rolling triage PR approach—where the agent appends new commits to an existing open triage PR rather than opening a new one each day—was the right call but had a subtle failure mode. We were detecting existing PRs by branch name, but the Managed Agents harness rewrites branch names into its own format. Three days of triage PRs stacked up on one repo before we caught it. The fix was to match on PR title instead, which the agent controls directly.

The distinction between a bug that should be fixed, a bug that should be documented but left alone, and a false positive turned out to matter a lot for the triage agent. The original version was too willing to defer—it would read a Sentry issue, decide it was "expected behavior," and move on without a fix. High-volume errors in particular need a different treatment: if a service is generating ten thousand of the same error per day, "expected behavior" isn't a complete answer even if each individual instance is technically expected. Volume is itself the problem.

Where We Go From Here

The feedback agent itself is a candidate for feedback-agent treatment. When the system makes a mistake—opens a PR with the wrong fix, misunderstands the scope of a request, fails where it should have succeeded—those failures are visible in the database and in the PR history. Feeding them back as training signal, or at minimum using them to improve the system prompt, is the obvious loop to close.

The architecture is also generic enough to cover domains beyond engineering. Any workflow where a queue of structured requests leads to a defined output that requires tool use and can be reviewed before taking effect is a candidate for this pattern. We're thinking about what else fits.