Sub-50ms Speech on One H100, at 2% of ElevenLabs' Price

TL;DR

Nari Labs, the small team behind the open Dia speech model, published a serving stack for Alibaba's Qwen3-TTS 12Hz 1.7B CustomVoice that holds sub-50 ms p95 time-to-first-audio through 10 requests per second on a single NVIDIA H100 SXM, and stays under 100 ms at 20 RPS. Both the serving implementation and the benchmark harness are Apache 2.0. At the $4.29 per hour they quote for a 1x H100 SXM instance, fully utilized, that lands at roughly $2 per 1M characters. Nari prices ElevenLabs V3 at $100 and Cartesia Sonic 3.5 at $49 for the same million.

They moved the goalposts, and they were right to

Most TTS latency numbers measure time-to-first-audio: request goes out, first bytes of PCM come back, stop the clock. Nari measures audible TTFA instead, defined as the time from dispatch to the first sample a human can actually hear.

The distinction is not pedantry. Their writeup notes that the first PCM a model returns often contains tens of milliseconds of quiet before any sustained sound. Under the old metric, a model that ships 80 ms of silence and then speaks scores identically to one that speaks immediately. Under the new one, it loses.

So they built a dynamic trimmer: detect sustained speech from short RMS windows, drop the samples before onset, stream the rest normally. It bought about 80 ms. The biggest single latency win in the entire post came from deleting nothing, faster.

Everyone is fast at 1 RPS. That is the trap.

Run any of these engines on an idle GPU with one client and you will get a number you like. Nari benchmarked four engines on the identical model and identical hardware, first at stock settings, and the stock numbers already tell you something.

vLLM-Omni posted the best default audible TTFA in the field at 277.883 ms, and stuttered on 100% of requests. VoxServe came in slower at 315.064 ms with zero stutters. SGLang-Omni took 1,140.69 ms and an engine they anonymize as M* took 1,159.956 ms, both clean.

The stutter is the part worth understanding. An underrun is what happens when playback catches up to the server: the client has drained its buffer and the next chunk has not arrived, so the stream gapes. Think of a bartender who slides you the first sip instantly and then disappears for the rest of the pint. Technically excellent time-to-first-drink. Terrible night out.

Once every engine is tuned to a no-underrun profile, the ranking inverts under load:

VoxServe goes from best in class at 49.3 ms to 363.2 ms at 6 RPS, a 7.4x degradation. vLLM-Omni, tuned, degrades by 1.6x. If you have been picking a speech engine off a demo on an empty GPU, this chart is the reason your production p95 looks nothing like the blog post you read.

The scheduler is the actual product

Qwen3-TTS is not one model doing one thing. Generation splits across three components: the Talker predicts the first codebook token for each audio frame, the Code Predictor generates the remaining 15, and a causal Codec turns codebook tokens into waveform samples.

Exposing all three as independently schedulable tasks is what lets the scheduler do the interesting part: deadline-aware prioritization. Requests that have not produced their first audio yet get high priority. Streams already playing become urgent only as they approach their next playback deadline, because until then their buffer is doing the waiting for you.

It is air traffic control for audio frames. A plane with fuel gets to circle; a plane running low jumps the queue. The scheduler picks an urgent request as an anchor and fills the rest of the batch with compatible work, which is how you get low latency without throwing away batching efficiency.

The rest of the stack is the usual grind, done properly: fixed CUDA graphs for the Code Predictor with a Triton attention kernel, state-cached incremental Codec decoding so each chunk does not re-decode history, and WebSocket input streaming so a speech-to-speech pipeline can start synthesizing before the upstream LLM has finished its sentence.

The money

Latency is the headline. Cost is the reason anyone will actually switch.

Their arithmetic: roughly 630 characters per second sustained at 10 RPS, on a box that costs $4.29 an hour. That is about 2.27M characters an hour, so a shade under $2 per million.

The vendor comparison is Nari's, so cross-check it. ElevenLabs publishes 6,000,000 credits for $990 on its Business plan and states that one text character equals one credit for its V2 multilingual models, which is about $165 per million at list. The same page notes API usage on newer models runs 0.5 to 1 credit per character, which brackets the low end near $82. Nari's $100 sits inside that range rather than under it. The gap to $2 survives either way.

What it actually costs you to run

The $2 number is a utilization bet, not a price. It assumes you keep an H100 SXM busy. At 10% duty cycle you are paying $20 per million characters and would have been better off with a card, and at 1% you are just heating a datacenter on someone else's behalf.

Other things the README will tell you before your credit card does:

• H100 or nothing. The published Docker image wants an NVIDIA H100, the NVIDIA Container Toolkit, a CUDA 13.0-compatible driver, and a Linux x86_64 host. This is not a laptop project.
• English, custom voices. The serving repo documents custom voices and English. The underlying Qwen3-TTS base model is broader, but broader is not what shipped here.
• It is early. 65 stars, 4 forks, Apache 2.0. You are an early adopter, with everything that implies about who fixes the first production bug.
• They benchmarked their own work. Nari builds speech products and picked the comparison set. The mitigation is that the harness is public, pinned to a dataset, and open-loop, so you can rerun it against your own text and call them on it.

That last point is the real story. An engine benchmark you can rerun is worth more than an engine benchmark you can only read, and open-sourcing the harness alongside the stack is a materially higher bar than most inference posts clear.

Key Takeaways

• Nari Labs open-sourced an Apache 2.0 serving stack for Qwen3-TTS 1.7B CustomVoice that holds sub-50 ms p95 audible TTFA through 10 RPS on one H100 SXM, and under 100 ms at 20 RPS.
• Measure audible time-to-first-audio, not first byte. Trimming leading silence bought about 80 ms with no change to inference speed.
• Single-client latency is a vanity metric. VoxServe leads at 49.3 ms at 1 RPS and collapses to 363.2 ms at 6 RPS; tuned vLLM-Omni goes 56.8 ms to 93.5 ms.
• Check for underruns, not just TTFA. vLLM-Omni's default profile had the fastest first audio in the field and stuttered on 100% of requests.
• Splitting Talker, Code Predictor, and Codec into independently schedulable tasks with deadline-aware priority is what buys latency without killing batch efficiency.
• Self-hosting lands near $2 per 1M characters only at high utilization on a $4.29/hour H100. Hosted vendors sit between roughly $49 and $165 depending on tier and model.

Sources: Nari Labs: How we made a text-to-speech model respond in sub-50 ms, nari-labs/nari-qwen3-tts, nari-labs/benchmarks, Qwen3-TTS-12Hz-1.7B-CustomVoice model card, vox-serve/vox-serve, vLLM: Engineering TTS Inference in vLLM-Omni, ElevenLabs pricing, Cartesia pricing