← Back to all posts
Tools

Superfluid Turns Mixed Machines Into One Local LLM Server, 10,656 tok/s on 12 Mac Studios

November 17, 2026 · 01:22 UTC · Tools
On this page 8 sections
Superfluid Turns Mixed Machines Into One Local LLM Server, 10,656 tok/s on 12 Mac Studios

TL;DR

Superfluid is an open-source LLM server from Base Compute that runs models on hardware you own. On one machine it schedules your chat ahead of agent work, so a prompt sent while eight agents run gets its first token in 1.5 seconds. Across machines it pools a fleet of mixed laptops, workstations and Mac Studios into a single endpoint that speaks the OpenAI, Anthropic and Ollama APIs. The technical paper published Nov 13 also documents a twelve-node Mac Studio cluster sustaining 10,656 tokens per second across 38,563 requests with zero failures.


At a glance
One server, your machines
1.47s
chat first token with 8 agents running
10,656
tok/s aggregate on 12 mac studios
38,563
requests in that run, zero failures
5,041
of 5,041 answered with one machine killed
3
runtimes: basert, llama.cpp, mlx
A-2.0
apache-2.0, rust, open api endpoints

What actually ships

Superfluid went public on Oct 16 and its paper followed on Nov 13. The repo is Rust under Apache-2.0, and the architecture is three processes: a daemon owns the APIs, chat templates, tokenizers, the scheduler and the session log; one worker process per loaded model owns the weights, KV cache and sampling; on other machines a node process serves the same model for the head.

Three runtimes plug in underneath: llama.cpp for GGUF files, MLX on Apple silicon, and BaseRT, Base Compute's own engine, for .base bundles. Nothing is vendored; each runtime installs from its own project's releases when a model id first needs it. One machine is two commands:

curl -fsSL https://basecompute.co/sf/install.sh | sh
superfluid serve basecompute/Qwen3.8-27B

That lands an OpenAI-compatible endpoint at 127.0.0.1:8453. The model id picks the runtime: a GGUF repo such as unsloth/Qwen3.8-27B-GGUF runs on llama.cpp, an mlx-community/... id runs on MLX, and Qwen3.8-27B in .base form runs on BaseRT. superfluid launch claude (or codex, opencode, cline, pi) points a coding agent at the server, and the Anthropic and Ollama compatible APIs are on the same port.

Your chat stops waiting behind the agents

Anyone who has run a local model while a coding agent works knows the failure mode: the agent fills the queue, you ask a quick question, and the server gets to you eventually. On the test rig, Qwen3.8-27B at 4 bits on an Apple M5 Pro with 64 GB, a chat sent behind eight busy agents got its first token in 1.47 seconds through Superfluid on llama.cpp. Plain llama-server took 291 seconds. Ollama took 380. On MLX the split was 2.01 seconds against 612 for mlx_lm.server. 291 seconds is long enough to google "why is my local model so slow", read three wrong answers, and switch to Ollama, which is worse.

Qwen3.8-27B 4-bit, Apple M5 Pro
First token of a chat sent behind 8 agents
1.47s
superfluid on llama.cpp
291s
llama-server
380s
ollama
2.01s
superfluid on mlx
612s
mlx_lm.server
Vendor benchmark, same device and model in all five cases. Source: Superfluid repo benchmarks and arXiv 2611.17753.

Three scheduler behaviors produce that number. Interactive requests preempt agent and batch work instead of queueing behind it. Requests that share a long system prompt reuse one prefill, so eight agents reading the same 4,000-token harness do not pay for it eight times. And every token is written to a durable log before it reaches the client, which is how a crashed worker restarts and the session continues where it stopped.

Shared prefill, lower is better
32 requests that share one 4k-token prompt
rival serverssuperfluid
llama.cpp family, gguf
sf llama.cpp
56s
llama-server
171s
ollama
88s
mlx family, apple silicon
sf mlx
418s
mlx_lm
907s
One shared prefill instead of 32. Source: Superfluid repo benchmarks, Qwen3.8-27B 4-bit on M5 Pro.

Kill the inference worker and Superfluid serves again in 1.75 seconds; llama-server and mlx_lm.server do not recover at all, and Ollama comes back in 1.83. The serving layer costs nothing in throughput on the common path: on Qwen3-4B on an M1 Max, Superfluid on llama.cpp matched llama-server at one concurrent request, 76.7 versus 76.8 tok/s, and ran 4 percent ahead at eight.

Fleet mode replicates instead of splitting

The design choice worth stealing is in the README in one line: the cluster replicates the model, it does not split it. Every node holds a full copy of the weights, the head places each session on one node, and only requests and tokens cross the network. Weights and KV caches stay home. Picture a kitchen where every cook knows the whole menu and the waiter gives each order to whoever is fastest, against six cooks each owning one station and every dish walking station to station. Replication buys availability and throughput, and it costs memory: every node must fit the whole model, so the fleet is capped by its smallest machine.

Fleet mode
How a pile of machines becomes one server
Head
superfluid serve
owns the endpoint and the fleet listener
Join
node join with token
each node holds a full copy of the model
Route
work follows speed
head places sessions by measured tok/s
Fail
a machine dies mid-stream
another node continues from delivered tokens
Links between machines are encrypted and only token holders can join. Source: Superfluid README.

Routing needs no per-machine configuration. In the mixed-hardware run, the two NVIDIA GB10s and the AMD Ryzen AI Max 395 each served 114 to 122 tok/s while an M4 Pro and an M6 carried 71 to 73, and the head simply sent more work where the throughput was. The kill test is the part you would not believe in a demo: six machines served 64 simulated office users for an hour while machines joined one at a time every six minutes, and at minute 42 one machine was killed mid-response and restarted eight minutes later. Run A, Qwen3-4B on llama.cpp across an M6, M4 Pro, M1 Max, Ryzen AI Max 395 and two GB10s, answered 5,041 of 5,041 requests. Run B, Qwen3.6-35B-A3B on BaseRT, answered 7,151 of 7,151. The 61 and 52 requests in flight on the killed machine all completed, because another node picked up each stream from the tokens already delivered.

Mixed fleet, 64 simulated users
Output tok/s as machines join, 1 to 6
one machinesix machines
run a, qwen3-4b on llama.cpp
1 machine
112
6 machines
534
run b, qwen3.6-35b on basert
1 machine
104
6 machines
476
Median first token fell from 98.5s to 11.7s in run A and 66.3s to 6.9s in run B. Source: arXiv 2611.17753.

One caveat before you wire the house together: the quickstart binds the head to 0.0.0.0 on two ports. The join flow is token-gated and the links are encrypted, but read the security documentation before putting a listener on a shared network.

Twelve Mac Studios, one endpoint

The top reference deployment in the README: twelve Mac Studio M3 Ultras with 512 GB each, serving Qwen3.6-35B-A3B on BaseRT, one of the twelve doubling as head. At 384 concurrent streams with 2,048-token prompts the cluster sustained 10,656 tok/s aggregate, 9,249 of it prompt processing and 1,407 output. Across six 12-minute load points it served 38,563 requests without a failure. Backed off to 192 streams with 512-token prompts, it pushed 3,072 output tok/s, 20 per stream, with a 2.0 second median first token. The launch post claims the fleet mode has run on up to 64 devices mixed across chips, generations and operating systems.

Reference deployment
Twelve Mac Studio M3 Ultras, one endpoint
10,656
tok/s aggregate, 384 streams
9,249
prompt tok/s, plus 1,407 output
38,563
requests across six load points
0
failed requests in the run
512 GB
per machine, full model copy
2.0s
median first token at 192 streams
Source: Superfluid README and arXiv 2611.17753.

Read the fine print

Every number above is Base Compute's own, from the repo's benchmark methodology and the paper, and nobody independent has reproduced them yet. The company sells local inference for a living, BaseRT is its engine, and the install script pipes from its own domain, so the project doubles as distribution even under Apache-2.0. The two community runtimes keep the scheduling layer, the interesting part, fully inspectable. The repo is also young: 31 stars and 16 open issues at time of writing. Benchmark on your own hardware before you point production at it.

Why you would run it

If your homelab is a Mac on the desk and an NVIDIA box under it, Superfluid gives that pile one address and one API, with each machine on the runtime that suits it. Chat preemption fixes the daily-driver problem: leave agents hammering the model and still get an answer when you type. Durable sessions mean the overnight job survives a worker restart, and a dead laptop costs its share of throughput instead of the whole endpoint. For anyone who already runs local models, the paper is also a decent checklist of what a local server should be doing for agent traffic: preemption, shared prefill, session logs, speed-aware routing.

Key Takeaways

  • Superfluid is an Apache-2.0 Rust LLM server from Base Compute, open-sourced Oct 16 with a technical paper on Nov 13 (arXiv 2611.17753), that serves one machine or a fleet through OpenAI, Anthropic and Ollama compatible APIs.
  • On one machine, Qwen3.8-27B at 4 bits on an M5 Pro: a chat behind eight agents gets its first token in 1.47s versus 291s for llama-server, 380s for Ollama and 612s for mlx_lm.server; 32 shared-prompt requests take 56s versus 171s.
  • Fleets replicate the model rather than splitting it: every node holds a full copy, the head routes sessions by measured speed, and a machine killed mid-response hands its streams to another node, which continues from delivered tokens. All 61 and 52 in-flight requests completed in the two kill tests.
  • A six-machine mixed fleet scaled from 112 to 534 tok/s (llama.cpp) and 104 to 476 tok/s (BaseRT) while answering 5,041 of 5,041 and 7,151 of 7,151 requests.
  • Twelve Mac Studio M3 Ultras sustained 10,656 tok/s aggregate across 38,563 requests with zero failures at 384 concurrent streams.
  • All benchmarks are vendor-reported and unreproduced, the repo is 31 stars young, and every node must fit the whole model, so the fleet is capped by its smallest machine.

Sources: Superfluid GitHub repository, Superfluid paper, arXiv 2611.17753, Base Compute launch post, Hugging Face post, Superfluid documentation, BaseRT repository

AISuperfluidLocal InferenceOpen Sourcellama.cppHomelabDistributed InferenceApache-2.0
CONSOLE
$