October 21, 2026 | AI | By Mahesh

DeepSeek has open-sourced DeepInfer, the inference framework its README says powers DeepSeek's large-scale production services, under the Apache 2.0 license [1]. It currently supports one model, DeepSeek-V4.1-Flash, on two kinds of hardware: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs [1]. AI Weekly's tracker places the release on October 20, and the README itself gives no date [2]. The repository ships offline inference only, so teams still need to build their own serving layer.

What DeepInfer is and what is new

DeepInfer is an open-source, high-performance inference framework for DeepSeek models. This release provides offline inference with the core features needed for serving at scale, and the default checkpoint is deepseek-ai/DeepSeek-V4.1-Flash-1017 [1]. That checkpoint was released on October 17 with MIT-licensed weights, a 1 million token context window and an API price of $0.30 per million input tokens at peak [5][6].

The release follows DeepSeek's September 30 move to open source six projects for Huawei's Ascend chips, among them DeepGEMM-Ascend, TileKernels, TileLang, DeepEP-Ascend, FlashMLA and DeepSelect [3][4]. DeepInfer builds on components such as FlashMLA, DeepGEMM and DeepEP and combines them into a model runtime [1].

Under the hood: parallelism, decoding and hardware

The runtime combines optimized kernels, a model and a sampler. For parallelism it uses attention data parallelism, balanced context parallelism and MoE expert parallelism with expert-parallel load balancing [1]. Speculative decoding uses draft tokens with confidence-scheduled verification, which the README calls DSpark, and bounded sliding-window attention replay reduces decoder prefill work. Scheduling is asynchronous by default [1].

Structured outputs can be constrained by regex, JSON schema or EBNF grammar. Batch-invariant inference and deterministic sampling are available on NVIDIA hardware only, which leaves a feature gap for Ascend users [1]. Building needs Linux, GCC 13 or later, Python 3.12 and either CUDA 13 for NVIDIA or CANN 9.2 for Ascend. A user sets the DEEPINFER_ACCELERATOR variable, runs ./build.sh and then python examples/offline_inference.py [1].

Evidence: what DeepSeek shows and what it leaves out

The README includes prefill and decode Pareto charts of throughput against latency and of output throughput against per-user speed, but the page text gives no numeric results [1]. The tests used 4 SM100 GPUs for prefill and 64 for decode, each inside one scale-up domain, and a run_sweep.sh script is included so others can reproduce them [1]. No independent measurements have been published, and the claim that DeepInfer powers production comes from DeepSeek.

There is also no comparison against vLLM or SGLang, the engines DeepSeek's model card points to for serving V4.1-Flash [5]. Earlier Ascend work reported roughly 90% to 95% of the physical bandwidth limit for expert-parallel dispatch at up to 32 ranks, with larger configurations still under optimization [3]. Those figures describe a different component and should not be read as DeepInfer results.

Pricing, availability and how to access it

DeepInfer is free under Apache 2.0 and lives in DeepSeek's GitHub repository. It builds from source in a clean Python 3.12 environment, and the model downloads on first run unless DEEPINFER_HF_MODEL_DIR points to a local copy [1]. Anyone who prefers a managed option can still call V4.1-Flash through DeepSeek's API, which costs $0.30 input and $1.20 output per million tokens at peak and half that off-peak [6].

Limitations, trade-offs and alternatives

The README says a complete inference service also needs a KV cache system, prefill and decode disaggregation and load balancing across data-parallel replicas, and none of those ship here [1]. The hardware needed is also large, since the decode benchmark used 64 GPUs, and DeepSeek lists only one supported model. Software maturity on Ascend remains a hurdle against Nvidia's CUDA ecosystem, and shared APIs reduce redevelopment without making the two environments interchangeable [3].

The star count shows strong interest, though the figure differs by source, with AI Weekly reporting 4.7 thousand and the repository page showing 12.4 thousand, so it is moving quickly [1][2]. For other open-weight options and their serving needs, see Depth Grid's coverage of Mistral Large 4 and of the Nvidia and Reflection AI talks.

Sources

1. DeepSeek. "DeepInfer," README, GitHub. github.com
2. AI Weekly. "AI News Today" launch tracker, October 21, 2026. aiweekly.co
3. Channel Insider. "DeepSeek, Huawei Expand Software Support for Ascend AI Chips," October 1, 2026. channelinsider.com
4. Let's Data Science. "DeepSeek Open-Sources Ascend Software With Huawei." letsdatascience.com
5. DeepSeek. "DeepSeek-V4.1-Flash-1017," model card, Hugging Face. huggingface.co
6. DeepSeek. "Models and Pricing," API documentation. api-docs.deepseek.com

Read More on Depth Grid

• Mistral Large 4 Preview: 1.05T Open-Weight MoE, Specs and Pricing
• Nvidia in Talks to Deepen Stake in or Buy Reflection AI
• Meta Muse Wonder Launch: 1M Context, $4 API Price, Benchmarks
• Claude Haiku 5.5 Launch: $0.10 Input, 1M Context, Benchmarks

Article by Mahesh | Depth Grid