Four Public AI Software Factories Share 12 Habits; Agent-Written Tests Add Nothing

TL;DR

I compared four public software factories (zazencodes, Eric Tech's super-board, Cole Medin's setup on Archon, and Antekeli's team benchmark) with our own. They agree on twelve habits, from the work item to the bounded repair loop. The public ones merged their first feature in minutes, an hour, or a day. The most useful evidence in the pile: on DeepSWE, banning the agent from writing its own unit and integration tests left success flat (65.3% to 66.2%, not significant) and cut spend by 9%.

Why I looked

We built our proofs before the first run: isolation drills, a judge that gets tested itself, and an attestation for every condition. Then I watched public factories with a fraction of that machinery merge a feature within a day.

So I sat down with what was public in October: zazencodes' "Software Factory for Claude Code", Eric Tech's super-board plugin, Cole Medin's ai-software-factory on top of Archon, the Factory.com docs, a long r/startups thread where a fractional CTO asked whether anyone had actually set one up, HumanLayer's "Why Software Factories Fail", and Kun Chen's DeepSWE test experiment. I wanted to know what they share, what they skip, and what I should copy.

Twelve things everyone does

They are more alike than I expected. They run on different hosts and keep state in different places, but the shape is the same:

• A work item with acceptance criteria is the unit of work. Issues, cards, Linear tickets or markdown jobs. Vague criteria are the first thing that breaks, so Eric lints them and Antekeli writes them as specs.
• Each lane is its own agent with a narrow instruction and fresh context, fed from files, never from the chat.
• State lives outside the conversation (a board column, a job.json, a ledger) and the run resumes from it.
• Repair is bounded. The builder gets findings back at most N times (two for zazencodes, configurable for Eric and Cole), then a person decides.
• Each job runs in isolation, usually a git worktree per feature, several in parallel.
• Merge policy is explicit. Auto-merge inside a size and risk envelope; money, auth, destructive schema changes and anything the approver is unsure of go to a human.
• Models are routed by size. Sonnet or Haiku for small work, Opus for large. Eric has a cheap grader decide; zazencodes fixes it per lane.
• Evidence rides on the work item: review files, screenshots, test reruns, traces.
• Kanban, not Scrum. Humans are the bottleneck on review, so the system's job is a short, well-formed human queue.
• Telemetry intake comes later, after the loop works (Sentry, PostHog, issues).
• Every one has a dashboard. In the simplest it is a single HTML file reading the job files.

The twelfth is the loop itself, and every build draws it the same way:

Where they split

The differences are in how much each one trusts its own judge, and how isolated the builder is.

A newer test adds more numbers at the fast end. Hash Kader at TechStackups built the smallest factory he could three times, in Herdr, T3 Code and Amp: GitHub issues in, one agent per issue in its own worktree, squash-merge, deploy to GitHub Pages. All three ran on free plans with Claude Opus 5.5 doing the work. Each got five issues live in under eight minutes (3.8, 5.3 and 7.7) with nobody touching the code. Then he opened two more issues, and none of the three noticed. They sat for ten minutes until he prompted the dispatcher again. Something still has to start the dispatcher, such as a GitHub Action or a scheduled script, unless you pay for Amp's hosted machines and webhooks. (Disclosure: Herdr is the terminal I run our own agents in.)

None of the public builds has a learning loop that changes the factory when a run fails. Eric's collector comes closest: it refiles recurring PR problems as new work. We have two tickets for it and no mechanism, so I can't throw stones. Paul Dix, who is building a factory for InfluxDB, argues this loop is where every factory ends up: some of its outputs are changes to the factory itself, so once agents take over the human steps it improves itself. He calls it recursive self-improvement. It's an argument: his post describes no mechanism and no run.

Tests the agent writes for itself

Kun Chen ran Sonnet 5.5 on DeepSWE: 111 tasks, 444 runs. In one arm the agent was banned from writing unit and integration tests. Success went from 65.3% to 66.2%, which is not significant. The no-tests arm took 6% less time and 9% less spend. On a 44-task subset, turning off test execution entirely changed nothing.

Tests written by the agent that wrote the code restate the implementation. They are the builder's own reading of the intent, checked against the builder's own reading of the intent. It is a student grading their own homework with the answer key they wrote.

What I take from it: the builder writes no tests of its own, and the judge has to be independent of the builder. That is the strongest outside support I've seen for the hidden stories and planted faults we built. The DeepSWE result covers unit and integration tests only. The evaluation also wrote 17 end-to-end tests, and the result says nothing about those. It also says nothing about regression suites guarding a long-lived codebase. Keep those. Stop demanding fresh unit tests as proof.

Dix gets to the same place another way. He ranks verification by cost: does it compile, unit tests, integration tests, stress runs that take hours or days, a production shadow of an affected customer on the new build, and a human last. Pick the cheapest step that gives real evidence. One of his example missions is to "remove all unit tests and ensure behaviors are verified through e2e tests" or on a larger testing platform. That's one practitioner's direction with no numbers behind it, and it points the same way as the DeepSWE run. He puts the human last on purpose: "Having a human check anything is the slowest and most expensive thing you can do." HumanLayer would put that human first, on the plan.

The rest of the evidence

• Review the plan, not the pull request. HumanLayer's argument: quality verification is far harder than tests passing, and models degrade a codebase without a human judge. A 30-minute plan review saves hours of code review, and vertical slices beat plans built in stack order.
• WIP limits are a feature. LeadSting's 24-hour autonomous run needed caps on open issues and open PRs to stay inside context and stay reviewable.
• Vague tickets get guessed. In the TechStackups run, "Homepage feels a bit plain" came back as three different designs, and all three tools gave the trail cards a colored edge by difficulty that no issue asked for. One agent also fixed an unrelated footer typo on the way past.
• The forge can switch you off. About an hour into the same test, GitHub suspended the tester's personal account after three new public repos, 21 issues, 19 agent-merged pull requests and about 40 Actions runs. GitHub gave no reason. His advice: use a dedicated account or organization and ramp up slowly, because a suspension also locks you out of every organization you belong to.
• Process theater is the failure mode. The skeptics in the thread have a point: when prompts replace verification, you get a factory that looks busy. The answer is enforced checks, not more prompts.

Antekeli's benchmark adds a cost data point. With the planner, reviewer and QA fixed on Opus, a cheap builder resolved 19 of 20 tickets.

Where we stand

Our own build, scored against the same field:

We built the referee first. Dix chose the same order on purpose: most of his focus this year has gone into test and verification infrastructure, because "a software factory can't work without proper verification." His post gives no first-build number, so it doesn't settle whether proofs-first pays off.

The plan from that review was short: fix every stop in the factory, never in the output. Cut the attestations to five (isolation held, hidden stories passed, the planted fault went red, the contract hash did not change, cross-family review happened). Stop the builder writing tests. Put a size cap and risk classes on auto-landing, WIP limits on each stage, and a retro step that files fixes against the template after a failed run.

Key Takeaways

• Four public software factories share the same skeleton: a work item with acceptance criteria, one narrow agent per lane, state outside the chat, bounded repair, isolation per job and an explicit merge policy.
• On DeepSWE, banning agent-written unit and integration tests moved success from 65.3% to 66.2% (not significant) and cut time 6% and spend 9%. The judge has to be independent of the builder.
• That result does not cover end-to-end tests or regression suites on a long-lived codebase. Keep those.
• Review the plan, cap work in progress, and enforce checks instead of adding prompts.
• A minimal factory on free tools shipped five issues in under eight minutes, but it won't pick up new work without a trigger, and one run got a personal GitHub account suspended within an hour.
• None of the public builds has a loop that fixes the factory itself after a failed run. Neither do we, yet.
• Ours differs mostly in isolation and the judge: a builder sandboxed with no network, hidden stories, a planted fault on every run, and review by a different model family.

Sources: zazencodes, "Software Factory for Claude Code" (video, README and BUILD_PROMPT.md); Eric Tech, super-board; Cole Medin, ai-software-factory on Archon; Factory.com, "Your Software Factory" docs; r/startups, "Has anyone here actually set up a software factory"; HumanLayer, "Why Software Factories Fail"; Kun Chen, DeepSWE evaluation of agent-written tests; Hash Kader, TechStackups, "Herdr vs T3 Code vs Amp: Software Factory Test"; Paul Dix, "The Software Factory and the Inevitability of Recursively Self-Improving Software".