A 104GB Model on a 48GB Mac. It Isn't Even the First Repo to Do It.

TL;DR

On September 1 Carlos López put slotstream on Show HN: a single Swift binary that runs Qwen3.8-Flash-Next, a 125B mixture-of-experts model that is 104 GB on disk at 4-bit, on a 48 GB MacBook Pro at about 12 tokens a second. It does that by keeping 3.8 GB of dense trunk resident and reading everything else off the SSD as the router asks for it. The thread's top comment was a list of five other repos doing roughly the same thing, and the author's own measurements say the interesting constraint was never RAM. It was the disk.

What actually ships

slotstream is MIT, has no Python at runtime, installs with one curl line into a folder in your home directory, and serves the Ollama and OpenAI chat API subsets on port 11434, so Open WebUI and the OpenAI SDKs work unchanged. The README's headline table for a 48 GB M5 Pro: about 12 tokens a second warm decode, a 2 second engine start because only the trunk loads, 32 GB peak memory, 104 GB of weights on disk. Releases are built by CI from the tagged commit with signed provenance you can check before running the binary.

The model is the one Alibaba published on August 26: 48 layers, 512 routed experts per layer with 10 routed plus 1 shared active per token, 6B active parameters out of 125B, plus a 51B parameter n-gram embedding table and a 4B multi-token-prediction head. slotstream pins the pipenetwork 4-bit MLX conversion, 103.8 GB across 24 files against 360 GB for the bf16 original.

Why the obvious approach dies

The lazy answer is to mmap the safetensors and let the kernel page in whatever gets touched. MLX cannot do that: it has no way to materialize part of a memory-mapped tensor, so a top-10 expert gather evaluates all 512 experts of the layer, the mmap path loads about 100 GB, and the process dies. The README reports that the stock mlx_lm loader took the 48 GB machine into 48 GB of swap without producing a single token, which is one way to find out how much swap you have.

slotstream's answer is a fixed pool of cache slots, each sized for one expert record, shared by all 48 layers. Experts are read with pread straight into a slot, and hot layers borrow slots from cold ones. The math never sees the pool: a gather over cached experts is bit-identical to a dense quantized matmul on the same weights, and greedy output is byte-identical between a 4 GB cache and a 24 GB one. Cache size changes speed, never output, and that equivalence is a standing test in the repo.

Picture a 48-floor hotel that owns one shared pool of rooms. A floor with a rush of guests takes rooms from a floor that is quiet, and nobody ends up on the wrong floor, because the router already said exactly which 10 experts this token needs.

Disk is the gate that bites first

The number the whole design rests on is in MEASUREMENTS.md, and the author admits two earlier attempts to measure it were wrong. The final method reads every offset at most once across 57 GB of real shards with the page cache and readahead disabled, so nothing can be served from cache. On the 2 TB Apple SSD in the M5 Pro, a cold random read of one 2.76 MB expert record hits 9.46 GB/s at queue depth 1 and 17.3 GB/s from queue depth 8 up. A 4 KiB read at queue depth 1 manages 0.08 GB/s.

That 100x gap is the quantitative case against page-granular anything. Fetching an expert as one 2.76 MB pread is carrying the whole binder off the shelf; page-sized IO is fetching it a sheet at a time and walking back for each one. It also produces a surprising ceiling: at 480 expert-uses per token and 17.3 GB/s, a cache with a zero percent hit rate still sustains about 13 tokens a second from IO alone. On this machine the binding constraint on small Macs is memory and compute, not bandwidth. The README's own caveat applies, though: base-storage MacBook Airs have much slower SSDs, and the smaller tiers have not been measured on real hardware.

What each Mac gets

Auto-sizing takes the lowest of three limits (33 GB, 70 percent of RAM, and 2 GB under the Metal working-set cap) and shrinks further while other apps hold memory, re-checking every 15 seconds and resizing the cache between requests. The 33 GB cap is the knee of the measured curve, not politeness: a gigabyte-at-a-time sweep from 34 to 84 GB found nothing that decoded or prefilled any faster, so a 128 GB Mac gets the same plan a 48 GB one does.

Prompt processing is the slow axis. All of it runs before the first token appears, so 8,000 tokens wait about a minute on the 48 GB machine and over three minutes at a 16 GB target, and prompt plus completion is capped at 32,768 tokens. Within a conversation you pay once, since follow-up turns prefill only what is new. The author got prefill from 40 to 92 tokens a second on the way here by raising tokens per pass from 256 to 2,048, because a prefill pass touches nearly every expert of every layer and re-reads the whole 68 GB set once per pass.

The draft head nobody converted

The 0.2.0 release, tagged the same day as the Show HN, adds speculative decoding using the model's own multi-token-prediction head, which predicts the token after next. The community 4-bit conversion dropped those tensors. slotstream restores them with a one-time conversion that pulls exactly the 31 needed tensors from the official release via ranged downloads, 4.9 GB in about two minutes, and writes a 1.47 GB draft-head file next to the weights. The Swift port of the head is bit-exact against the MLX Python reference.

Measured first-draft accept rate is 85.8 percent, 3.5 accepted tokens per verify round at depth 4. The honest part is where it does not help: the head costs 1.6 GB, and at a 16 GB target the A/B came back at 0.96x, so auto mode keeps it off below roughly 26 GB of target, where that memory buys more expert cache. The 1.5x to 1.9x expected above that is arithmetic from the accept rate, and the changelog says the large-cache A/B is still pending.

The fifth repo problem

The top comment on the Hacker News thread named mlx-moe-offload, streamlx, mlx-moe, mlx-flash, and deepseek-v4-flash-mlx and asked why the local-LLM ecosystem needs every idea rediscovered five times and wrapped in a new README. The author, who says they have been doing open source since 2014, answered that they would add a benchmark and comparison table.

The comment has a point and misses one. mlx-flash (130 stars, last pushed in June) streams dense weights layer by layer with a prefetch controller, which is a different problem. deepseek-v4-flash-mlx and ds4-ssd are the closest cousins: LRU or slot-bank expert paging for DeepSeek V4-Flash, a roughly 100B MoE with 256 experts and about 13B active, and the former reports 4.5 to 5 tokens a second on a 48 GB Mac. Different model, twice the active parameters, so not a benchmark. Meanwhile the mlx-lm feature request for exactly this, opened June 27, is still open with no maintainer reply. Five repos is what an unfilled hole in the platform looks like.

Caveats before you burn 110 GB of disk

• One machine. Every number is from a single M5 Pro with 48 GB and a 2 TB SSD. Smaller tiers are estimates, and the README says so.
• Quality of the pinned weights. The pipenetwork model card puts the uniform 4-bit build at perplexity 5.39 against 4.47 for bf16, and says the mixed 4/8-bit build (106.2 GB, within 0.013 NLL per token of bf16) is the one to use at this size. slotstream v0 runs exactly one model name, and it is the uniform one.
• Disk first. 110 GB free is the floor, so a 512 GB Mac is the realistic minimum regardless of RAM.
• The download. Through 0.2.0, pull ran on one TCP connection whatever the flag said, because HTTP/2 multiplexed every request over it: 25 to 40 MB/s from home. 0.2.1 opens eight real connections and measured 112 MB/s from a gigabit datacenter link (16 minutes) and 50 to 63 at home. At 25 Mbps plan on about nine hours.
• API gaps. Tools, images, JSON-schema output, and logprobs return a 400 rather than being silently ignored. The Ollama CLI could not connect in 0.2.0; 0.2.1 fixes it.
• macOS 14 and 15 have only had the installer exercised, not the runtime. The dev machine runs macOS 26.

Key Takeaways

• slotstream decodes a 104 GB, 125B MoE at about 12 tokens a second on a 48 GB M5 Pro by keeping 3.8 GB resident and streaming 68 GB of experts plus a 32 GB n-gram table from SSD.
• Expert-sized 2.76 MB reads hit 17.3 GB/s cold on the Apple SSD; 4 KiB reads get 0.08 GB/s, which is why page-granular mmap loses and a record layout wins.
• Cache size changes speed, never output: a 30-of-512 expert cache produced byte-identical greedy text to a 181-of-512 one, at 7.3 GB peak.
• Speculative decoding with the model's own draft head measured an 85.8 percent accept rate, but it is a net loss below roughly 26 GB of target and stays off there.
• At least four other repos do a version of SSD expert streaming on Apple Silicon, and the upstream mlx-lm request for it has sat open since June 27.
• The pinned uniform 4-bit weights are measurably worse than the mixed 4/8-bit build the same publisher recommends.

Sources: GitHub, carloslfu/slotstream (README), slotstream MEASUREMENTS.md, slotstream CHANGELOG.md, slotstream v0.2.0 release notes, Hacker News, Show HN thread, Hugging Face, Qwen/Qwen3.8-Flash-Next model card, Hugging Face, pipenetwork/Qwen3.8-Flash-Next-MLX-4bit, Qwen blog, Qwen3.8-Flash-Next, GitHub, mlx-lm issue #1438, GitHub, matt-k-wong/mlx-flash, GitHub, ssd-moe/deepseek-v4-flash-mlx, GitHub, Anemll/ds4-ssd.