Nolan Lawson Benchmark: Agents Build in Hono+htmx for $2.67 and Angular for $4.97 per Run
TL;DR
Nolan Lawson, a Seattle programmer at Socket, built an open benchmark that asks a question most framework debates skip: how many tokens, minutes and dollars does a coding agent burn building the same app in each framework, and how fast is the result? He pointed Claude Opus 5.5 at one spec, one test suite and one prompt, five times per framework, across 12 stacks. Hono+htmx was the cheapest at a median $2.67 per build. Angular was the most expensive at $4.97, took twice as long and produced the most code. Astro shipped zero kilobytes of JavaScript and the fastest page. The harness, agent-framework-benchmark, and the full results are public, and the post hit the front page of Hacker News with more than 200 comments arguing about what it proves.
The setup
The app is a Pokédex backed by a local copy of PokéAPI: search, filters, and list pages for types, moves and game data. Lawson calls it a single "holotype," borrowing Jason Miller's term, and says it sits closer to Wikipedia than to Figma. Each agent runs alone in a fresh Podman container with 4 CPUs and 8 GB of RAM, sees only the exported template (spec, local API, tests and design files), and gets one line of instructions: implement the app in framework X, use the latest stable version, work until the tests pass.
The model was Claude Opus 5.5 at high effort through Claude Code 2.1.310, the run config records. Where a framework publishes official agent tooling, the agent got it: the Angular CLI MCP server and skills, Astro's docs MCP, Nuxt's MCP, the Svelte Claude Code plugin, the React Router skill, Next's devtools MCP and the Hono skills. SolidStart, Qwik, Gracile and vanilla got nothing because nothing official exists. Lawson's README treats that unevenness as part of the measurement: it is what you get when you pick the framework.
The acceptance tests query the accessibility tree rather than DOM structure, so an agent cannot pass by matching markup. They also rerun the finished app against an API with changed data, which catches apps that bake Pokémon in at build time. Static pre-rendering of data was banned outright, otherwise one architecture wins by skipping the API. Finished apps were served behind Caddy and measured with Lighthouse on mobile settings with 4x CPU throttling and Slow 4G. The README itself opens with a warning that the project is AI-generated, so this is a benchmark of agents that agents largely wrote.
What a build costs
Costs are Claude Code's reported API-equivalent prices (Lawson is on a Max plan, so he did not pay them per run). The medians across five runs, which we checked against the summary.json in the results repo, line up exactly with the post:
Time tracks cost. Astro finished in a median 8.2 minutes and Hono+htmx in 8.6, while Angular took 17.2 and ranged up to 23.7. Tokens tell the same story at the other end of the table: Nuxt used 3.85 million tokens per run including cache reads, Angular 9.76 million. Two of Angular's five runs also failed to produce a measurable app (one rejected requests from anything but localhost, despite the spec), so its page metrics rest on three runs while its cost figures include all five.
The vanilla result is the one that stings a particular HN faction. A plain Vite project with no UI framework came fourth on cost, behind Nuxt, and was slower to build than Astro, Hono+htmx and Nuxt. Lawson's reading is that a framework can help an agent more than raw DOM code does, which is roughly the argument frameworks have made about humans for a decade.
Where the tokens went
Lawson read the transcripts, and the expensive stacks got expensive for different reasons. SvelteKit agents read the SvelteKit 3 migration guide in every run, because version 3 is too new for the model to know. Gracile cost tokens because the agents had never heard of it, and in two runs they quietly skipped Lit altogether. Angular just wrote more: 2,009 non-blank lines against Hono+htmx's 1,370 for the same passing tests, which is either a verbosity tax or enterprise-grade thoroughness depending on who signs your paycheck.
That line count matters beyond the first build. Every line the agent writes is luggage it has to re-check on every future flight: each later edit means reading the file again, so code size turns into a recurring token bill rather than a one-time one. Lawson puts it as "the code size ultimately impacts the number of tokens, including in future edits."
Official tooling was used erratically. The SvelteKit agents called their MCP in five of five runs and the React Router agents loaded their skill every time, but the Astro agents touched their MCP in only two runs, Nuxt's MCP was never called at all, and Angular's agents used the MCP in three runs and the skill files in two. Lawson draws a sharp conclusion from that pattern: "I have a strong suspicion, for example, that the current crop of MCPs and skills may be doing more harm than good." Anyone who spent last quarter wiring up MCP servers may want to sit down for that one.
What the agents shipped
Page-load results split cleanly by architecture. Multi-page apps that render on the server shipped almost no client JavaScript and posted the best Core Web Vitals. Single-page apps shipped the most and paid for it in Total Blocking Time and Largest Contentful Paint.
Largest Contentful Paint followed: Astro 810 ms, Gracile+Lit 956, Hono+htmx 978, Qwik 979, up to TanStack Start at 2,028 and Next.js at 2,090. Qwik is the odd one out: an SPA that scores like an MPA. Total Blocking Time ran from 0 ms for five frameworks to 71 ms for Next.js and 103 ms for Angular. Lighthouse performance scores were all 97 to 100 and accessibility scores all 100, which Lawson notes are saturated and tell you little.
The caveats, which Lawson lists himself
This is one app type, one model at one effort level, one browser engine, and Lighthouse on the home page only. The Pokédex is content-heavy, which favors server-rendered MPAs on first load, and the README adds a detail the post glosses: the MPA servers call the API at localhost speed while the browser's calls get throttled, so SPAs carry a handicap on these metrics. Nothing after page load is measured. An earlier version timed a scripted search flow, but the numbers mostly reflected debounce settings and harness overhead, so it was cut. Variance is large: Lighthouse sometimes picked a different element as the LCP between runs, Next.js's LCP range (1,028 to 2,187 ms) comes from one app that skipped lazy-loading images, and Lawson's own rule is that one iteration is an anecdote and three is the minimum.
The HN thread pushed on the obvious seam. Several commenters argued that a form-heavy or realtime app would reshuffle the table, and one called the whole exercise evidence that "vibes" beat benchmarks for tool choice. Lawson's defense, near the end of his post, is that he set out to show framework decisions can still rest on measurements in the agent era. He says outright that it is not meant as the final word on which framework a vibe-coder should choose. The post closes by asking whether you pick a stack on evidence or on what you already know, and the thread answered mostly with the latter.
Run it yourself
The harness needs Linux, Podman 4.3 or later, Node 20 or later, about 15 GB of disk and Claude credentials. One command reproduces the published run:
node bench/run.mjs --frameworks=all --iterations=5 --effort=highThe Pokémon data (about 2 GB from pinned PokéAPI commits) downloads once into a base image, so no agent ever spends tokens fetching it. Model, effort, Claude Code version, CPU and memory limits are all flags, so swapping in a cheaper model or a competitor's agent is a one-line change. Lawson's advice: read bench/run.mjs first, since it is the only thing that runs outside a container, and use an API key with a spending limit, because the agent and anything it installs can read the credential. Pull requests are closed. He does not plan to keep running new frameworks forever and would rather see forks run on other people's tokens.
Key Takeaways
- Across 60 Opus 5.5 runs, the same app cost a median $2.67 to build in Hono+htmx and $4.97 in Angular, with Astro, Nuxt and vanilla Vite under $3.
- Angular builds took 17.2 minutes, 9.76M tokens and 2,009 lines; Hono+htmx took 8.6 minutes, 4.26M tokens and 1,370 lines for the same passing tests.
- Server-rendered MPAs shipped 0 to 18 KiB of JavaScript and the fastest LCP; Next.js shipped 142 KiB and loaded slowest at 2,090 ms.
- Code size is a recurring token cost: every future edit re-reads what the agent wrote, so the verbose stack keeps charging you.
- Official MCPs and skills were used inconsistently (Nuxt's never, SvelteKit's always), and Lawson suspects the current crop may do more harm than good.
- Results are for one content-heavy app, one model, and page load only; the harness is open and a single command reruns it.
Sources: Nolan Lawson, "The AI cost of JavaScript frameworks", agent-framework-benchmark (GitHub), agent-framework-benchmark-results (GitHub), Hacker News discussion