Glonce

Nari Labs hits sub-50ms TTS latency on a single H100

infrastructureThe Glonce editorsHacker News🔥63Glonce rating63Impact60Novelty50Relevance85Surprise45Players45Credibility82
Nari Labs hits sub-50ms TTS latency on a single H100

Nari Labs, a Y Combinator-backed startup previously known for the open Dia text-to-speech model (over two million downloads, ranked #1 on Hugging Face), published a blog post detailing a custom serving implementation for Qwen3-TTS 1.7B CustomVoice, a popular permissively licensed TTS model. On a single NVIDIA H100 SXM, the system sustains 10 requests per second (RPS) while keeping p95 time-to-first-audio (TTFA) under 50 milliseconds, and stays below 100 ms even at 20 RPS. At 10 RPS it produces about 630 characters per second, which on a $4.29-per-hour H100 rental works out to roughly $2 per 1 million characters at full utilization. Nari Labs compares this to ElevenLabs V3 at $100 per 1 million characters and Cartesia Sonic 3.5 at $49 per 1 million characters, noting Cartesia's TTFA is higher than its own.

The team benchmarked five implementations under identical conditions: their own, vLLM-Omni, SGLang-Omni, VoxServe, and M*, using five-minute runs of Poisson open-loop traffic (a methodology borrowed from Fireworks AI's LLM benchmarking approach), reconstructing playback from streamed PCM audio and verifying it with Deepgram speech-to-text. After each rival engine was tuned for latency, continuity and capacity, only Nari Labs' own build held sub-50 ms p95 TTFA through 10 RPS. VoxServe reached sub-50 ms only at 1 RPS; the other three engines did not reach it even there. By around 6 RPS, every other engine's p95 TTFA had risen to roughly 100 ms or worse. SGLang-Omni's upstream support was still incomplete at the time of testing, in mid-August 2026.

The speed comes from re-architecting how the model's three stages are scheduled. Qwen3-TTS generates audio through a Talker (predicts the first codebook token per frame), a Code Predictor (generates the remaining 15 codebook tokens per frame, always in a fixed 15-step sequence) and a causal Codec (converts codebook tokens to waveform). Most existing implementations split serving into two stages, running Talker and Code Predictor together and the Codec separately. Nari Labs instead exposes all three as independently schedulable tasks under one shared scheduler, an approach the post says draws on the M* design (arXiv). The scheduler treats any request that has not yet produced its first audio as high priority, while an already-streaming request only becomes urgent as it nears its next playback deadline; it then batches an urgent 'anchor' request together with compatible work to keep GPU utilization high.

Three further engineering changes contribute: because the Code Predictor always runs exactly 15 steps per frame, Nari Labs preallocates its KV cache and captures the entire frame-generation loop as a single CUDA graph, paired with a Triton attention kernel tuned for its short, bounded context. The Codec is rebuilt around cached Transformer and convolutional state so that, after an initial full decode of the first audio, later chunks are generated by decoding only newly arrived frames instead of replaying the whole utterance history. Separately, the team also trims leading silence from generated audio (worth about 80 ms of TTFA on its own), captures CUDA graphs for a set of predefined batch sizes and splits any larger cohort across scheduling turns, defers GPU termination checks to avoid unnecessary CPU-GPU synchronization, and supports streaming text input so a speech-to-speech pipeline can start synthesizing before an upstream LLM finishes generating its full response. Nari Labs has open-sourced both the serving implementation and the benchmark, and says Qwen3-TTS is the first step toward broader multimodal inference work covering image, video and world models.

Key facts

  • Nari Labs' Qwen3-TTS 1.7B CustomVoice implementation sustains 10 requests per second with sub-50ms p95 time-to-first-audio on a single NVIDIA H100 SXM, staying under 100ms even at 20 RPS.
  • At 10 RPS the system produces about 630 characters per second, working out to roughly $2 per 1 million characters on a $4.29-per-hour H100 instance, versus $100/1M for ElevenLabs V3 and $49/1M for Cartesia Sonic 3.5 (at a higher TTFA).
  • Benchmarked against vLLM-Omni, SGLang-Omni, VoxServe and M* under five-minute Poisson open-loop traffic runs; after tuning, only Nari Labs' own build held sub-50ms p95 TTFA past 1 RPS, and every rival engine's p95 TTFA reached roughly 100ms or worse by around 6 RPS.
  • The gains come from unifying the model's three stages (Talker, Code Predictor, Codec) under one shared scheduler inspired by the M* design, a fixed-step CUDA graph for the Code Predictor, and a state-cached Codec that decodes only newly arrived audio frames.
  • Nari Labs, backed by Y Combinator and previously known for the Dia TTS model (over two million downloads, #1 on Hugging Face), open-sourced the serving implementation and the benchmark.

Why it matters

Conversational and speech-to-speech AI products live or die on how quickly a reply becomes audible, whether playback then runs without gaps, whether that holds as more requests arrive, and whether the speech stays intelligible. Nari Labs' result shows that a serving-layer redesign, not a new model, pushed those measures well past existing engines on an already-open, permissively licensed TTS model, and dropped the per-character cost to a fraction of what commercial APIs like ElevenLabs charge. It is a demonstration that open models can be engineered to match or beat proprietary paid services on latency and cost when the serving stack, not just the model, gets the same attention.

Who it affects

Teams building real-time voice agents, speech-to-speech pipelines, or any product where a user is waiting live for synthesized audio stand to benefit most directly, since TTFA and cost per character are exactly the levers this work targets. It also matters to infrastructure engineers designing multimodal serving systems generally, since the scheduling techniques (unifying heterogeneous model stages, deadline-aware batching, state-cached incremental decoding) generalize beyond TTS. Companies currently paying ElevenLabs or Cartesia for high-volume TTS have a concrete cost comparison to weigh against self-hosting.

How to use it

Nari Labs says it has open-sourced both the serving implementation and the benchmark suite; the source does not give a repository link, hosted product, pricing tier, or licensing terms for using the code directly, only the cost estimate for running it: about $2 per 1 million characters on a single $4.29-per-hour H100 SXM instance at full utilization. Anyone adopting it would need to run Qwen3-TTS 1.7B CustomVoice specifically, and would need to replicate the scheduler and codec-caching design described in the post rather than getting it as a drop-in configuration change to an existing engine.

How solid is it

This is a vendor-published benchmark: Nari Labs is reporting its own implementation's performance against four alternatives it configured itself, with no independent third party involved. The methodology is disclosed in detail, five-minute Poisson open-loop traffic runs modeled on Fireworks AI's LLM benchmarking approach, audio reconstructed from streamed PCM and checked with Deepgram speech-to-text, which makes the comparison reproducible in principle. Nari Labs itself flags two limits: SGLang-Omni's upstream support was still incomplete at test time in mid-August 2026, which may have disadvantaged that engine, and the post concedes that despite its tuning effort, some configuration of a rival engine might still edge out its own results.

Risks and caveats

The headline cost figure, about $2 per 1 million characters, explicitly excludes networking, idle capacity and operational overhead, so it is a best-case per-GPU number rather than a fully loaded production cost. No audio-quality metric, such as a MOS score, is given for any of the five engines, so the speed and cost comparison says nothing about whether output quality is comparable across them. The source does not say whether this serving stack is already running in a production, paying-customer deployment, only that the implementation and benchmark have been open-sourced.

“The key is not merely splitting them into parts, but bringing all three onto a shared scheduling surface managed by one scheduler.”

— Nari Labs, engineering blog post