An iOS Dev Ran a 26B Model in 2 GB of RAM, and It's Not a Party Trick

TL;DR

TurboFieldfare, a new open-source Swift and Metal inference engine by iOS developer Andrey Mikhaylov, runs Google's Gemma 4 26B-A4B in about 2 GB of resident RAM on any Apple Silicon Mac, including the base 8 GB machines Apple spent years insisting were plenty. The trick: keep the model's 1.35 GB shared core and KV cache in memory, and stream the mixture-of-experts weights off the SSD per token. An 8 GB M2 MacBook Air decodes at 5.1-6.3 tok/s, an M5 Pro at 31-35, and the Hacker News thread that pushed it past 600 points includes an independent reproduction rather than a debunking. Apache 2.0, 14.3 GB on disk, one very specific model.

The 8 GB Mac just became a 26B-class machine

The standard rule for running language models locally is brutal and simple: the whole model must fit in memory, so your RAM is your ceiling. TurboFieldfare attacks the rule directly. It is not a wrapper around MLX or llama.cpp; it is a single-purpose engine, written in Swift with custom Metal kernels for quantized GEMV, attention, MoE routing, RoPE, and sampling, tuned for exactly one checkpoint: the instruction-tuned Gemma 4 26B-A4B from Hugging Face.

The installer never even materializes the original checkpoint. It streams the required byte ranges from Hugging Face and repacks them directly into its own on-disk layout: about 14.3 GB at 4-bit (MLX affine, group 64, with an 8-bit router). At inference time, resident memory sits around 2 GB. That is less RAM than your browser is currently spending on tabs you forgot about.

The trick: an MoE model never needs most of itself

This only works because of what Gemma 4 26B-A4B is. Per the Gemma 4 docs and model card, it has 25.2B total parameters but routes each token through just 8 of 128 small experts plus one shared expert, about 3.8B active parameters per token. A mixture-of-experts model is a hospital with 128 specialists on staff: any given patient only ever sees eight of them. You do not need the whole staff in the building, you need the right eight paged in fast enough that nobody notices the wait.

So TurboFieldfare keeps the parts every token touches (attention, embeddings, the shared expert, the router) plus an FP16 KV cache pinned in RAM, and treats the SSD as the warehouse for the other 128 experts. A 16-slot LFU cache exploits the fact that expert choice is sticky: the author measured roughly 40% of experts repeating on the next token and 57% recurring within two. Prefill batches up to 128 tokens per chunk so one fetched expert serves many rows at once.

Why mmap was not enough

The obvious objection, raised immediately on Hacker News: llama.cpp has memory-mapped weights for years, so what is new? The author's answer is that mmap left the machine mostly idle. His mmap prototype managed about 0.5 tok/s; switching to explicit bounded parallel pread calls got the same setup to 4 tok/s before further optimization. The difference is that page faults fetch small pages lazily, one stall at a time, while the router tells you the full list of needed experts up front, so the engine can issue large reads and overlap them with the GPU's attention work for that layer.

The hardware trend is doing the rest. The author measured about 83 ms to read an expert on an M2 Air but 12 ms on an M5, which is most of the gap between the Air's 5-6 tok/s and the M5 Pro's 31-35. At the low end this is a storage benchmark wearing a language model costume: an M4 Mac mini with 16 GB, tested independently in the thread, lands at just over 5 tok/s, right beside the older Air.

The numbers, honestly

The tax for that memory cut is real, and the project does not hide it. Commenters put conventional MLX inference of the same checkpoint at about 75 tok/s using 14 GB of RAM on the same M5 Pro, more than twice the streamed speed. If your Mac has the memory, load the whole model and be happy. The point of TurboFieldfare is the machines that do not.

The caveats section

Keep your expectations calibrated. Gemma 4 26B-A4B activates 3.8B parameters per token, so you are getting a very good sparse model, not a dense 26B; the README itself warns the model can still repeat itself or get things wrong. The published 2 GB figure covers the weights plus a 4K-context KV cache, and longer contexts grow that resident set. The engine is deliberately single-model: this one Gemma checkpoint, nothing else, and only on Apple Silicon.

The skepticism in the thread was about practicality, not honesty: whether a fanless Air throttles on long sessions, and whether you want your SSD servicing a bounded read storm for every token. Notably, nobody who actually ran it disputed the throughput claims.

Why this matters beyond one Mac app

Every consequential open-weight drop of the past month is a sparse MoE, and they keep getting bigger than any RAM you can buy: Kimi K3 alone is a 1.4 TB download. Expert streaming is the only architecture-honest way those models ever touch consumer hardware, and TurboFieldfare is a working, Apache-licensed existence proof that the residency floor is the shared core, not the parameter count. The measured expert locality (40% next-token reuse) is the load-bearing fact, and it is a property of MoE routing in general, not of this app.

It also quietly rewrites the local-AI shopping spec. The gap between 5 tok/s on an M2 Air and 31-35 on an M5 Pro tracks storage latency (83 ms versus 12 ms per expert), not GPU grunt. Apple charges by the RAM tier, but for this class of workload the number to check is how fast the SSD can hand back a few megabytes of expert, over and over, forever.

Key Takeaways

• TurboFieldfare, an Apache 2.0 Swift/Metal engine, runs Gemma 4 26B-A4B in about 2 GB of resident RAM on any Apple Silicon Mac, storing 14.3 GB of 4-bit weights on SSD.
• It works because the model is a mixture-of-experts: only 8 of 128 routed experts (about 3.8B of 25.2B parameters) fire per token, and the engine streams just those, with a 16-slot LFU cache exploiting roughly 40% next-token expert reuse.
• Explicit parallel pread calls overlapped with GPU compute beat mmap by roughly 8x in the author's tests (4 tok/s versus 0.5) because the router announces needed experts before they are used.
• Reported speeds: 5.1-6.3 tok/s on a base 8 GB M2 MacBook Air, 31-35 tok/s on an M5 Pro, with an independent M4 Mac mini run confirming the low end; in-RAM MLX on the same M5 Pro does about 75 tok/s using 14 GB.
• The technique generalizes to the current wave of giant open MoE models, and it makes SSD latency, not RAM size, the spec that decides what your machine can run.

Sources: TurboFieldfare on GitHub, Hacker News discussion, Gemma 4 26B-A4B model card, Google Gemma 4 documentation