← Back to all posts
News

Ankit Gupta's Coding Agents Speed FPGA LLM Inference 33x to 213 Tokens/s

November 21, 2026 · 04:17 UTC · News
On this page 9 sections
Ankit Gupta's Coding Agents Speed FPGA LLM Inference 33x to 213 Tokens/s

TL;DR

Ankit Gupta, a general partner at Y Combinator and former CTO of Reverie Labs, published "Vibecoding an inference chip" on Nov 20. He pointed a multi-day coding-agent loop at an AWS F2 FPGA and had it implement, then optimize, inference for DeepGrove's ternary-weight model. The first full bit-exact image ran at 6.43 tokens per second; the final one runs at 213.13, a 33.13x speedup, with all 24 benchmark turns matching a pinned CPU reference bit for bit. The loop wrote roughly 3 million lines of code, parsed 30 million lines of experiment output, and burned about $10,000 in Claude and OpenAI tokens. Gupta had never written a line of Verilog.


At a glance
An agent loop, an FPGA, zero hardware background
33.1x
speedup from the first bit-exact image
213
tokens per second on the final image
0
lines of Verilog Gupta had written before
~3M
lines of code the agents wrote
$10k
in Claude and OpenAI tokens, whole loop
24/24
benchmark turns bit-exact vs the CPU reference

What was actually running

The hardware is an F2 instance from AWS, with 16 GB of HBM attached to the FPGA; Gupta had Codex book the reservation because he had never experimented with FPGAs before. The model comes from DeepGrove, whose ternary mixture-of-experts architecture is built for memory-starved machines. The speculative draft model is a cut-down of Maple, DeepGrove's open-weights 20B-parameter, 1B-active ternary LLM, running on the host CPUs while the FPGA verifies its guesses.

The driver was a cloud agent in Conductor, running for days across hundreds of experiments, with OpenAI's GPT-6 Astra and Anthropic's Opus swapping in and out; both cleared his "smart-enough threshold," in his words, and he used them interchangeably. His own input reads like coaching, not engineering. Of 166 prompts in the session, seven steered, and the samples he reproduces are "you're a superintelligent model, time to be more creative" and "let's go for 2x faster, you got this." Twice the ML background surfaced: a nudge toward speculative decoding, and a "why is that running on CPU, you should port it."

The rule that made autonomy possible

One constraint holds the whole project together: every candidate build must reproduce the reference model's outputs bit for bit on an 8-chat, 3-turn benchmark. The final image scores 24 of 24 turns exact. When the grader is "same bits as the CPU reference," a wrong change fails and reverts, and there is no room to hack the score. That is what lets someone who cannot read Verilog leave an agent alone overnight with a memory subsystem.

Ternary weights help on the math side. At 2 bits, a weight stops multiplying: it drives a mux that picks x, ~x or 0, so the arithmetic collapses into routing and adds. A three-way light switch where a multiplier used to be, and an FPGA is mostly switches.

How the loop worked
One experiment, every few minutes, for days
Propose
Agent drafts a change
C++ and Verilog, no human review
Screen
Lint, unit tests, quick synthesis
logic-depth check, hour-long sims skipped
Verify
Outputs must match the CPU reference
same bits, or the change reverts
Measure
Run the benchmark
tokens per second logged per image
Compound
Keep what wins, iterate
14 kept steps turn 6.43 into 213.13 tok/s
The fast screen let two hangs through that only RTL simulation would have caught; those cost 38 hours on the board.

The loop also learned what not to run. Cycle-accurate simulations and full place-and-route take hours, so on advice from the DeepGrove founders it settled into a short loop of lint, unit tests, quick synthesis and a logic-depth screen, saving place-and-route for integration. A soak test caught random faults at a 231.25 MHz kernel clock, so the final image runs one step lower at 225. Simulation results, timing signoff and board results were kept in separate buckets throughout, and the post never claims a number from the wrong one.

Fourteen steps, no thunderbolt

The optimization log is the most copyable part of the writeup. Fourteen recorded steps, each worth roughly 1.1x to 1.7x, compounding into 33.13x. Host-side command overhead dropped from about 10 ms to 0.9 ms. A 128-lane attention engine started reading the KV cache in bursts instead of a word at a time. Speculative decoding arrived twice, first with prompt-lookup drafts, then with the cut-down Maple doing the drafting. Lossless 5-trit packing shaved 20 percent off weight bytes. The last step alone, a head loader that reads both of its memory banks at once, was worth 17.3 percent, measured on and off on a single image.

decode speed, measured
Modest steps, compounding
tok/s on the benchmark
baseline
6.4
host
8.1
attention
14.0
glue engine
30.2
pipeline
49.2
head shadow
94.4
5-trit pack
171.3
final
213.1
Selected steps from the 14-step optimization log. Source: ankitg.me.

If you have not met speculative decoding: a cheap model guesses a few tokens and the real model checks all of them in one pass, a junior dictating the memo and a senior initialing the whole paragraph at once. Wrong guesses cost one pass, right ones come out several tokens at a time.

Long context holds up too. In a single generation run out to the full 8,192-token context, early images managed 37 tokens per second in the 6-8k range; the last two hold 76-77 and finish the whole run in about 78 seconds, down from 162.3.

The scoreboard, prices attached

The comparison runs all come from Nov 15 on AWS, at the on-demand prices recorded with the experiment. The final image on an f2.6xlarge ($1.98/hour) serves 213.13 tok/s. An exactly matching CPU implementation on 32 Zen 5 cores ($1.95/hour) serves 129.5, so the FPGA is 1.65x faster than a CPU at effectively the same price. The GPU rows are the humbling ones: a community CUDA port on a $0.53/hour T4 reaches 166.3 tok/s, and DeepGrove's own GPU-offload fork reaches 35.7, but they match 0 and 1 of 24 turns exactly. The post calls both "pretty close in practice" while conceding they fail the bit-exact criterion, which is the honest way to present a table where your rig does not win every row.

same 8-chat x 3-turn benchmark
Against the alternatives
cpu, bit-exactfpga, bit-exact
f2 fpga
213.1
cuda t4
166.3
cpu 32t
129.5
cpu 24t
99.6
t4 fork
35.7
Navy rows are T4 GPU ports that fail the bit-exact bar (0/24 and 1/24 exact turns). On-demand prices: $1.98/hour f2.6xlarge, $1.95 32-core m8a, $0.53 g4dn. Source: ankitg.me.

Did the human advice matter?

Midway through, the DeepGrove founders sent a 4,741-character prompt distilling lessons from their own batch-1 FPGA decoder. Gupta pasted it in on Oct 29 at 21:50 UTC, then later asked the agent to audit its own logs for what it had actually used. Within 70 seconds of receiving the prompt, the agent had sorted it into ten ideas to adopt and five to reject, and its retrospective says its biggest design ideas were already in the plan before the paste.

All five numerics shortcuts got rejected, lower-precision activations, a deferred norm, an 8-bit KV cache, online softmax and 16-bit probabilities, because each one changes output bits and the bit-exact rule outranks speed. The process advice stuck immediately: stop a simulation that had been running for 24 hours, skip two 250 MHz builds, keep the dev loop short. One real find did come out of the exercise, AWS's own reference memory design, which sustains 426 GB/s across all 32 channels and became the memory subsystem.

So the division of labor in the log is clear, if a little bruising for the humans. The agents had the design ideas covered; the most durable human contribution was telling them what not to run.

Read it with these caveats

  • Every number is author-measured on an author-chosen benchmark, and the comparison configurations never got the same optimization loop, a gap the post admits.
  • No repository is public yet; "share the code" is among the first requests in the Hacker News thread (122 points, 49 comments inside nine hours).
  • It is one instance family and one small context window. Gupta says plainly that he would not run production inference this way; his claim is about what a small team can now attempt, and thread regulars argue both sides of FPGA economics at scale.

For calibration, this is not the first agent-written inference engine: one commenter in the same thread posted his own, built with an earlier Opus. His line is the right summary: "implementing transformers is the new 'hello world' of AI assisted coding."

What to steal

  • Bit-exact oracles unlock unfamiliar domains. If a reference implementation grades every change as same-bits-or-not, you can let agents rewrite code you cannot read.
  • Pick graders that cannot be argued with. A score can be gamed; a bit-for-bit diff against a pinned reference cannot.
  • Bank the 1.5x steps. Fourteen modest wins compounded to 33x, and none of them was a breakthrough on its own.
  • Spend your human attention on iteration policy. The highest-leverage input in the whole log was "stop running things that take hours."
  • Keep your evidence buckets separate. Simulation, timing signoff, board results: never claim from the wrong one, and soak-test your clocks before trusting a speedup.

Key Takeaways

  • On Nov 20, YC general partner Ankit Gupta published a full account of a multi-day coding-agent loop (Conductor, GPT-6 Astra and Opus used interchangeably) implementing and optimizing LLM inference on an AWS F2 FPGA, despite never having written Verilog.
  • Throughput rose from 6.43 to 213.13 tok/s across 14 logged steps, 33.13x total, with all 24 turns of the 8-chat, 3-turn benchmark bit-exact against a pinned CPU reference; full-context runs fell from 162.3 to about 78 seconds.
  • The loop wrote about 3 million lines, parsed 30 million lines of output, ran hundreds of experiments over 166 prompts (seven steering), and cost roughly $10,000 in Claude and OpenAI tokens on a $1.98/hour instance.
  • Price-matched, the FPGA beats a 32-core Zen 5 CPU twin by 1.65x; T4 GPU ports are cheaper and fast (166.3 tok/s) but match only 0/24 and 1/24 turns exactly.
  • The agent's audit of a 4,741-character expert prompt found most design ideas already in its plan and rejected all five numerics shortcuts under the bit-exact rule; what stuck was process advice, at the cost of two board-only hangs and 38 hours.
  • Caveats: author-measured, author-chosen benchmark, no public repo yet, one instance family, and an 8,192-token context model the author himself would not put in production.

Sources: Vibecoding an inference chip, Ankit Gupta, Hacker News thread, ankitg.me, DeepGrove Maple-Preview model card, AWS F2 instances, Conductor

AIFPGACoding AgentsLLM InferenceAnkit GuptaDeepGroveAWS F2Speculative Decoding
CONSOLE
$