Their Rust Rewrite Went 48 Requests to 33. It Was a Win.

TL;DR

Experiential is an Apache-2.0 LLM gateway that landed on the Hacker News front page on August 27 with 208 points. The interesting part is not the product, it is the benchmark doc the team published alongside it: their compiled Rust data plane tops out at roughly 33 requests per second, while the Python engine it replaced hit 48. They shipped it anyway, because at 128 concurrent streams the Python engine's median time to first byte was 7.2 seconds and the Rust engine's was 25 milliseconds. The throughput ceiling was never the language. It was six fsynced SQLite writes per request.

What actually shipped

The repo went public under Apache-2.0 with 711 stars and 59 forks as of this writing, first commit dated June 24, 2026. It is a single OpenAI-compatible endpoint (plus the Anthropic Messages API) fronting OpenAI, Anthropic, Gemini, Azure, Bedrock, Fireworks, and OpenRouter, with per-identity spend limits and a SQLite ledger that accounts every token. Install is pip install experiential, then exp.

On paper that is LiteLLM with a nicer wizard, and the top HN comment asked exactly that. The team's answer was the optimizer: point it at your production traces and it fits a router, then optionally fine-tunes an open-weight model you own on the same traffic. Pass-through on your own keys is free at list price with 0% markup; the intelligence layer is the paid enterprise tier. So the honest framing is that the gateway is a loss leader for a router product, which is fine, and at least they say so in the thread.

The rewrite that lost throughput

Here is the number that makes this worth reading. The team benchmarked both engines on one 16-vCPU Azure VM against a near-instant mock upstream, load generator and gateway pinned to disjoint cores. The results are in the repo:

Rewriting a hot path in Rust and coming out 31% slower is the kind of result most teams quietly drop from the launch post. This one put it in a heading.

Why the ceiling was fsync

Both engines write the same ledger. Every request through the gateway commits roughly six fsynced SQLite transactions at synchronous=FULL, and until those durably land, the request is not accounted for. The native engine pays all six plus three GIL crossings to hand control back to Python, which is precisely why it plateaus below the pure-Python engine.

The mental model: you hired a sprinter to replace a jogger on a delivery route, and both of them still have to stop at the same six checkpoints and wait for a stamp. The sprinter does not finish first, and if he has to hand the clipboard through a window three times, he finishes slightly last.

The team says so plainly in the doc: scaling requests per second past roughly 30 to 50 per process is a durability-architecture question, not a language question. Live-provider ladders reproduce the same ceiling on both engines.

Where the rewrite actually paid

Throughput was never the problem. Concurrent streams were. The Python engine parses JSON, validates a pydantic event, counts bytes, and re-encodes SSE for every single delta, all on one event loop behind the GIL. It saturates around 2,000 output tokens per second per process and its executor admits only 64 requests at a time.

The consequences are ugly and specific. With a mock upstream streaming 200 tokens at 40 tokens per second, the Python engine's median time to first byte is 938 ms at 64 concurrent streams and 7.2 seconds at 128, because every stream past the cap waits out an entire upstream generation. The 5-second streams themselves stretch to a 12.6-second median. The native engine holds a flat ~25 ms median first byte through 128 concurrent streams with undistorted 5.28-second generations, and relays roughly 21,000 tokens per second per process.

Against real providers at low concurrency the same gap shows up in the overhead the gateway adds on top of a direct call:

Soak runs put about 24,000 live streaming requests through each provider arm at a steady ~31 requests per second with error rates at or below 0.02%. The bench ledgers closed at 189,316 requests and 27.1 million tokens with zero accounting drift, which for a billing path is the number that actually matters.

The part that is not a gateway

The optimizer is the reason this exists. You feed it OpenTelemetry traces from your current agent, exp build turns them into a simulation, and exp optimize router fits a routing policy against held-out tasks. Then exp optimize model hands the router's traces to Tinker, the LoRA fine-tuning API from Thinking Machines Lab, and trains an open-weight model you own. There is a public trace dataset on Hugging Face if you want to try the loop without wiring up your own telemetry.

The published evidence run is small and deliberately so: 100 normalized traces produce 50 fit tasks and 20 sealed held-out tasks, expanding to 140 candidate-task cells of which 10 are observed and 130 simulated, then 140 persisted judgments under a ceiling of 200. Observed hosted-provider spend for the whole run: exactly $0.00, because the simulation is text and the clients are injected. The online quality signal is an LLM judge with a rubric the user calibrates by hand in a TUI.

The finding worth stealing

Buried in the HN thread is the most useful thing here, from the poster answering a question about prompt caching. Switching models constantly destroys your cached input tokens, so the router mostly does not:

The trick is to rarely switch, or switch at task boundaries. Often the conclusion of routing is actually "this one model is actually at the pareto front for this task, just use it always".

He also notes the router tunes reasoning effort, not just model choice, and that Opus 5 at low reasoning frequently matches Opus 5 at high reasoning for a given task. If you run agents at any volume, that second observation is worth an afternoon of measurement on your own traffic, whether or not you install any of this.

Caveats

• The benchmarks are self-run. One 16-vCPU Azure VM, the team's own harness, mostly against a mock upstream. Reproduction commands are published, which is more than most launches offer, but nobody outside has replayed them yet.
• The repo is two months old. First commit June 24, 2026, and the team says it pivoted to the gateway only recently. 711 stars and 59 forks is early interest, not a battle-tested deployment record.
• The docs and the founder disagree on telemetry. The README says anonymous PostHog product telemetry is enabled by default; in the thread the poster says it is off by default. Either way, exp config telemetry disable exists, and the stated exclusion list covers prompts, traces, paths, model names, and credentials.
• The dashboard in the README is hosted-only. A commenter discovered this the awkward way. The intelligence features (per-prompt optimization, caching, the model trained on your traffic) are the enterprise product, not the Apache-2.0 repo.
• It is a crowded lane. LiteLLM, Bifrost, and vLLM's semantic router are all chasing the same slot, a point raised in the thread more than once.

Key Takeaways

• A Rust data-plane rewrite took the gateway from ~48 to ~33 requests per second, because the bottleneck is six fsynced SQLite commits per request, not interpreter speed.
• It was still correct: median time to first byte at 128 concurrent streams went from 7.2 seconds to a flat ~25 ms, and per-process relay went from ~2,000 to ~21,000 tokens per second.
• Before you rewrite a hot path, measure whether your ceiling is CPU or durability. If it is fsync, a faster language buys you nothing and may cost you throughput.
• The router's own conclusion is usually "stop switching models": constant switching wrecks prompt caching, so pick one model per task boundary and tune reasoning effort instead.
• Apache-2.0, pip install experiential, 0% markup on BYOK pass-through, with the trained-on-your-traffic layer reserved for the paid tier.
• Publishing the benchmark where your rewrite lost is a credibility move, and rarer than it should be.

Sources: experientiallabs/experiential on GitHub, native gateway data plane benchmarks, W16 router and sandbox evidence, Show HN discussion, Experiential Labs pricing, Thinking Machines Lab Tinker, wmo-terminal-tasks-traces dataset.