A 14MB Agent Runs on an ESP32. BFCL Says Don't Get Excited.

TL;DR

On August 10, Cactus Compute shipped Needle 2, a 45M-parameter tool-calling model baked into a single self-contained 14MB binary that runs a full session in roughly 28MB of RAM. It decodes at 500 tokens per second on a Raspberry Pi 5 and reaches microcontrollers. On the Seal-Tools out-of-domain split it beats models five to six times its size. On BFCL v4 it finishes dead last of the four models Cactus benchmarked. Both things are true, and which one matters depends entirely on what you are building.

What actually shipped

Needle 2 is not a chat model that happens to be small. It is a model that solves every problem as a function call, and that constraint is the whole design. You declare tool schemas, it emits calls. Structured extraction is the same operation with the record shape declared as the only tool.

The artifact is the interesting part. The weights are compressed to what Cactus calls CQ2-bit and baked directly into the executable, so there is no runtime to install, no checkpoint to download, and no network call at startup. You get one file. Your app's JavaScript bundle is probably bigger.

Builds ship for ARM64, x86-64, ARMv7, RISC-V, MIPS32el and WebAssembly, across macOS, Windows, Linux, Android, iOS, watchOS and tvOS. Python users get pip install cactus-needle. The code is MIT on GitHub; the weights carry Apache 2.0 on the Hugging Face model card.

Cactus reports the model reaching microcontrollers like the ESP32-P4, and notes that others have run it on an ESP32-S3 in about 11MB. Reported decode speeds: 500 tokens per second on a Pi 5, 400 to 1,500 on VR headsets including the Meta Quest 3S and Apple Vision Pro, and 300 to 700 on sub-$200 phones like Samsung's A-series.

The architecture: throw out the MLP, keep the attention

Needle 2 is built on what Cactus calls a Simple Attention Network, and there is an actual paper behind it. A Controlled Study of Attention-Only Transformers (Ndubuaku et al., July 20, 2026) asks a blunt question: do transformers need feed-forward layers at all? Controlling for parameters, compute and depth from 6M up to 87M parameters and 105B tokens, the team found that ripping out the FFN costs 0.47 nats. Spend those parameters on more attention layers instead, and the gap shrinks to 0.006 nats.

Think of the FFN as the model's private scratchpad, separate from the part that decides what to look at. The paper's finding is that you can burn the scratchpad and hand its budget to the looking, and almost nothing is lost.

Needle 2 goes further and swaps the dense up-and-down projections for a Hadamard MLP: a fixed Walsh-Hadamard transform with no learned weights to store or read. Around that sit grouped-query attention, hashed n-gram "engram" tables acting as key-value memory, and four residual streams routed by learned gates, in a 27-layer, 512-wide stack. The result is 70 MFLOPs per token against 87 for a parameter-matched transformer and 540 for FunctionGemma 270M.

The contract is unusually strict

Three design choices do most of the reliability work here, and they are worth stealing even if you never run this model.

• Grammar-constrained decode. A byte-level grammar is compiled from your JSON schemas, including ranges, patterns, lengths and enums. The model literally cannot emit a value that violates the schema. There is no separate JSON mode because there is no unconstrained mode.
• Bounded memory. A 256-token sliding window with the tool definitions pinned as KV sinks, so peak RAM stays near 28MB no matter how long the conversation runs. Long-session memory growth, the thing that kills embedded deployments, is simply not on the table.
• Tool retrieval built in. Declare more than five tools and a contrastive head embeds every schema once, then renders only the top five per turn, rebuilding the grammar over that subset. An unselected tool is unreachable, not just unlikely.

And when nothing fits, the model returns the empty call []. There is no free-text fallback, no apologetic paragraph, no improvisation. For a thing wired to your door locks, refusing to answer is a feature.

That confidence score is the escape hatch. Every response carries one, computed as the minimum of a post-hoc calibrated head scoring the full prompt plus the emitted call, and the raw decoding probability of the call tokens. Both signals have to agree. You pick a threshold, act above it, and route below it to a bigger model in the cloud. It is a bouncer who has to check two IDs before letting anything through.

Now the benchmarks, honestly

Cactus published four evaluation sets and did not hide the bad one, which earns some credit. The results split cleanly by task shape.

On Seal-Tools, a multi-call tool-learning benchmark, Needle 2 wins outright: 32.6% accuracy in-domain against 26.9% for LFM2.5 230M and 16.3% for FunctionGemma 270M, and 28.7% out-of-domain against 17.0% and 15.6%. On Mobile Actions (961 rows) it lands at 63.7%, a hair behind FunctionGemma's 64.0% and LFM2.5's 69.1%, but with by far the best function-name accuracy at 98.3%. It knows which tool to reach for; filling the arguments is where it drops points.

Then BFCL v4 single-turn, 3,641 rows, the benchmark most people actually cite: Needle 2 scores 42.6%. FunctionGemma 270M gets 46.1%, LFM2.5 230M gets 60.8%, Apple's on-device foundation model gets 61.7%. That is not a rounding error, and "trades wins with models 5x to 70x larger" is doing real work in that sentence.

One number holds up across both: well-formed output rate. Needle 2 emits syntactically valid calls 93.4% of the time on BFCL, within a point of LFM2.5 and Apple FM, which is what you would expect from grammar-constrained decoding. It is not producing garbage. It is producing clean JSON that is sometimes the wrong clean JSON.

What the thread found

The Show HN hit 178 points and 138 comments in a day, and the criticism was consistent: out-of-distribution prompts in the web playground produce nonsense tool calls. Several commenters reported the demo picking bizarre functions for off-topic input. The mitigating detail is that the confidence score came back at zero on those, which is exactly the case the escalation contract exists for, but it also tells you the base model is not going to work well on your tools without fine-tuning.

Cactus is upfront about that. The pitch includes an automated data-generation pipeline and end-to-end trainability, with fine-tuning on a Mac or PC measured in minutes to hours. Needle 2 is a base you specialize, not a drop-in general assistant. Version 1, a 26M model released May 12 and distilled from Gemini, was explicitly a distillation; this one was trained from scratch.

Who should care

If you are building a cloud agent, none of this matters. Route to a frontier model and move on.

If you are putting an agent on hardware where there is no reliable network, no thermal headroom, and a bill of materials that laughs at the idea of a GPU, this is a different conversation. A 14MB binary with a 28MB RAM ceiling and 70 MFLOPs per token fits places nothing else fits. The BFCL gap is the price of that, and for a fixed set of a dozen tools you fine-tuned yourself, the general benchmark was never the right question.

Key Takeaways

• The artifact is the story. A 45M-parameter model as one 14MB self-contained binary with no runtime, no download and no network puts agentic tool calling on hardware that has never had it.
• Attention-only is a real result, not a gimmick. The arXiv study shows removing feed-forward layers costs 0.47 nats, and reallocating that budget into attention closes the gap to 0.006 nats.
• Read the benchmark split before you believe the headline. First place on Seal-Tools out-of-domain, last place on BFCL v4 at 42.6% against 61.7% for Apple FM.
• Grammar constraints and a calibrated confidence gate are the transferable ideas. Schema-compiled decoding plus a two-signal confidence floor gives you escalation instead of wrong actions, at any model size.
• Plan on fine-tuning. Out of the box the model is flaky on prompts outside its training distribution, and Cactus positions it as a base you specialize on your own tool set.
• Licensing is clean. MIT code, Apache 2.0 weights, builds down to RISC-V and MIPS32el, so nothing about deployment is legally awkward.

Sources: Cactus Compute: Needle 2, Needle 2 model card on Hugging Face, cactus-compute/needle on GitHub, arXiv:2607.18363, A Controlled Study of Attention-Only Transformers, Cactus Compute: Needle v1, Show HN discussion, Berkeley Function Calling Leaderboard, Seal-Tools benchmark