DeepSeek Open-Sources DeepInfer, Its API Inference Engine, With Huawei Ascend Support

TL;DR

DeepSeek has published DeepInfer, which its README calls "the high-performance inference framework that powers DeepSeek's large-scale production services," under the Apache 2.0 license. The release supports one model, DeepSeek-V4.1-Flash, on two kinds of hardware: Nvidia SM100 GPUs and Huawei Ascend 950 NPUs. You get the kernels, the MoE parallelism, speculative decoding and the scheduler. You do not get the KV cache system, prefill/decode disaggregation or load balancing across replicas, which are the parts that turn an engine into a service. The repo passed 13,000 GitHub stars within a day of hitting Hacker News, where the launch thread drew 2,142 points and 836 comments.

What DeepSeek actually shipped

The repository was created on October 16, and DeepSeek announced it on October 20. The README credits its lineage plainly: DeepInfer "was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM." In practice it is a heavily customized vLLM descendant tuned for one model family.

The feature list:

• Kernels: DeepSeek's own FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, plus custom TileLang kernels.
• Parallelism: attention data parallelism with balanced context parallelism, and MoE expert parallelism with EPLB (expert-parallel load balancing).
• DSpark with confidence-scheduled verification: draft tokens speed up decoding, and confidence estimates decide when to verify them. The default maximum draft length is 5.
• SWA bounded replay: the decoder's prefill replays only a bounded suffix of the context instead of the whole thing. The default V4.1 setting keeps 140 tokens.
• Asynchronous scheduling: CPU batch scheduling overlaps with GPU execution, on by default.
• Batch-invariant inference and deterministic sampling: identical outputs regardless of what else shares the batch, and identical outputs for identical inputs with a fixed seed. Both are Nvidia-only for now.
• Structured outputs: regex, JSON schema or EBNF grammar constraints.

Batch invariance deserves a second look. On an engine without it, the numbers your request produces can shift slightly depending on which strangers' requests got batched alongside it, because the kernels reduce in a different order. Think of a cashier whose total on your groceries changes by a cent depending on who is standing behind you in line. DeepInfer can pin that down, which matters if you run evals, cache outputs, or debug "why did the agent do something different this time."

The missing pieces

DeepSeek is upfront about the gaps. The README lists three components "required for a complete inference service" that are absent: a KV cache system, VE/prefill/decode disaggregation ("most required engine functionality is already implemented"), and load balancing across data-parallel replicas for both prefill and decode. What ships today is offline inference.

The KV cache is the big omission. Prefix caching is where a lot of real-world serving savings come from, and the system that stores and serves those prefixes across a cluster is exactly what's missing. Getting DeepInfer without it is like getting a Formula 1 engine without the fuel system: it will run on the dyno, but your pit crew still has some building to do.

DeepSeek has also published deepseek-recipe, an MIT-licensed repo that, according to The Decoder, replicates the official API's token processing so self-hosted deployments behave like the hosted one. The Decoder also reports DeepSeek's statement that code for future DeepSeek models will ship as open source.

The numbers DeepSeek published

The README includes two Pareto charts for DeepSeek-V4.1-Flash with a 131,072-token context, plus a sweep script to reproduce them. Prefill was measured on 4 SM100 GPUs and decode on 64, each inside one scale-up domain. The values below are read off DeepSeek's charts and rounded, so treat them as approximate.

Decode ran with 64-way expert and data parallelism, three draft tokens per step, and 75% per-token draft acceptance. The trade-off curve is the classic one: pack more requests per GPU and total throughput climbs while each user's stream slows down.

Prefill ran with 4-way expert and data parallelism and the 140-token bounded replay. Non-cached input throughput climbs from about 60,000 tokens/s per GPU at a 4,096-token batch (around 70 ms per step) to about 90,000 tokens/s at 65,536 tokens (around 730 ms), with most of the gain arriving by 16,384.

What the README does not include is a head-to-head against stock vLLM or SGLang on the same hardware. That was the most-asked question in the Hacker News thread, and it is still open. Interconnects calls DeepInfer "likely far more efficient" than those tools for DeepSeek's models and ranks it among the bigger open-source AI artifacts of recent months, but "likely" is doing some work in that sentence until someone runs the numbers.

Huawei gets equal billing

The supported-hardware list has exactly two lines: Nvidia SM100 (Blackwell datacenter parts) and Huawei Ascend 950. The Ascend path needs CANN 9.2 and pairs with the DeepEP-Ascend and DeepGEMM-Ascend libraries DeepSeek released in late September. In DeepSeek's production stack, a Chinese accelerator gets the same billing as Nvidia's newest datacenter GPUs.

The timing is pointed too. The release landed the same day the DHS order excluding DeepSeek from US civilian agency systems became public. US agencies can't run DeepSeek's products, but anyone else can now run its serving stack.

Who this is for

If you run a 3090 shelf in the basement, this release is not for you. The build wants Linux, GCC 13 or newer, Python 3.12 and CUDA 13, the published benchmarks use 4 to 64 SM100 GPUs, and the only supported checkpoint is V4.1-Flash. Hopper and consumer cards are not on the list.

The real audience is inference providers. Anyone reselling DeepSeek tokens now has the vendor's own kernels, scheduler and speculative-decoding setup to start from, instead of reverse-engineering them from papers. Interconnects expects quick adoption by providers, which would push DeepSeek prices down across the ecosystem or widen provider margins. For the rest of us, the useful artifacts are the techniques: confidence-scheduled draft verification, bounded SWA replay, and a batch-invariant mode you can study under an Apache license.

Key Takeaways

• DeepInfer is DeepSeek's production inference framework, open-sourced under Apache 2.0 and descended from vLLM.
• It supports only DeepSeek-V4.1-Flash, on Nvidia SM100 GPUs and Huawei Ascend 950 NPUs.
• The KV cache system, prefill/decode disaggregation and replica load balancing are not included, so it is an engine, not a drop-in API service.
• DeepSeek's charts show about 26,000 decode tokens/s per GPU at roughly 100 tokens/s per user, and up to about 90,000 prefill tokens/s per GPU, with no published vLLM or SGLang comparison.
• Batch-invariant inference and deterministic sampling (Nvidia only) make identical prompts return identical outputs regardless of batch neighbors.
• The main beneficiaries are inference providers, and DeepSeek token prices may fall further as they adopt it.

Sources: DeepInfer on GitHub (README, eval and benchmark charts), deepseek-recipe on GitHub, The Decoder, Interconnects, Hacker News discussion, DeepSeek-V4.1-Flash-1017 on Hugging Face