DeepSeek Open-Sourced Its Production Inference Stack: What DeepInfer Gives You, and What It Doesn't

DeepSeek did something on October 20 that no frontier lab has done before: it open-sourced the inference framework that actually runs its production API. Not a research demo, not a stripped-down reference implementation — DeepInfer is the engine behind DeepSeek's live traffic, released under Apache 2.0 on GitHub, together with a technical report that discloses the cluster's real throughput and revenue numbers. Nine days later it has 21,000+ stars, and half of Chinese AI Twitter is arguing about gross margins.

The numbers DeepSeek disclosed with the code

The technical report accompanying the release contains two figures nobody outside DeepSeek had real data on before:

• 155 trillion tokens per day — the service throughput of a single DeepSeek cluster running V4.1 Flash.
• $38 per GPU per hour — overall revenue per card, calculated at off-peak electricity prices. That's roughly 254 RMB.

Those two numbers come from DeepSeek's own technical report, as reported by Yicai (第一财经) — I couldn't pull the original report text at publish time, so treat the exact figures as reported-through-a-press-source. What happened next is the interesting part: third parties took the disclosed revenue figure and reverse-engineered margins. One calculation circulating on NetEase arrived at 82% gross margin; Taiwan-based coverage put it above 95%. Both can't be right, and honestly neither is authoritative — the margin depends on assumptions about GPU capex, electricity, and staffing that DeepSeek didn't publish. My take: the revenue-per-card number is the solid datum, the margin estimates are parlor math until DeepSeek confirms a figure itself.

For context on what $38/GPU-hour means: an H100 rents for around $2–4/hour on the open market. Even granting that SM100-class hardware and the full cluster overhead cost far more than a rental, the gap is the whole story — DeepSeek is extracting roughly an order of magnitude more revenue per card than generic cloud GPU pricing, because inference efficiency, not model quality alone, is where the money is.

What's actually in the repo

First, the honest labeling: DeepInfer was built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM, per the README's acknowledgments. DeepSeek didn't write an engine from scratch — they took vLLM and pushed it through their production grinder. The repo was created October 16 and hit GitHub publicly on October 20; it crossed 21,000 stars within nine days (I pulled the count live from the GitHub API for this article).

The feature list is where the production-earned details show:

• Parallelism stack — Attention DP with balanced CP, MoE expert parallelism with EPLB rebalancing. This is the large-scale-serving machinery, not single-GPU conveniences.
• DSpark with confidence-scheduled verification — DeepSeek's speculative decoding setup, with a configurable draft length (default 5 tokens).
• Batch-invariant inference and deterministic sampling — bit-identical outputs across batch compositions, NVIDIA-only behind DEEPINFER_BATCH_INVARIANT=1. This matters more than it sounds: reproducibility has been a persistent pain in high-throughput serving.
• SWA bounded replay — decoder prefill work is cut by replaying only a bounded suffix, 140 tokens kept by default for V4.1 settings.
• Structured outputs — regex, JSON schema, or EBNF grammar constraints passed directly through SamplingParams.
• Asynchronous scheduling — CPU batch scheduling overlapped with device execution, on by default.

The kernels underneath are DeepSeek's existing open-source pieces — FlashMLA, DeepGEMM, DeepEP, plus custom TileLang kernels — so DeepInfer is best understood as the orchestration layer that ties the stack DeepSeek has been releasing all year into one coherent engine.

What it deliberately leaves out

The README is refreshingly explicit that this is not a complete serving stack. To run a full inference service you still need to bring your own: a KV cache system, prefill/decode disaggregation, and cross-replica load balancing. The engine-side functionality for disaggregation "is already implemented" in the framework, but the glue is on you.

That framing matters for anyone who saw "open-sourced their production stack" headlines and assumed plug-and-play. DeepInfer gives you the hardest single component — the batch scheduler and kernel orchestration that produces those throughput numbers — and leaves the surrounding service architecture as an exercise. For a team already running vLLM or SGLang at scale, that's a valuable donation. For everyone else, it's a masterclass to study, not a product to install.

The hardware wall

Here's the part that most Chinese-language coverage downplays: DeepInfer supports exactly two hardware targets — NVIDIA SM100 GPUs (that's Blackwell B200/GB200 class, not anything you own) and Huawei Ascend 950 NPUs. Build requirements: Linux, GCC 13+, Python 3.12, CUDA 13 or CANN 9.2.

No H100. No H200. No A100. No consumer GPUs. The default checkpoint is DeepSeek-V4.1-Flash-1017 on Hugging Face, and the reference benchmarks ran prefill on 4 SM100 GPUs and decode on 64 — each within a single scale-up domain. That 64-GPU decode figure tells you what class of deployment this framework is designed for.

The Ascend 950 support is arguably the bigger story than the benchmarks. A frontier lab's production inference engine shipping with first-class domestic-NPU support is a statement about where Chinese AI infrastructure is heading, and it means the framework is a real migration path for operators outside NVIDIA's ecosystem — if they can get 950s, which is its own supply-chain question.

What it means if you just use the API

If you're not self-hosting anything, why care? Because this release is DeepSeek showing its work on cost. The company has consistently priced its API below what the market assumes its costs must be, and the disclosed $38/GPU-hour revenue figure — paired with an open engine — is a public argument that current prices are sustainable. We've covered DeepSeek's off-peak pricing before; the economics only work because the serving layer squeezes this much throughput out of each card.

Practical implications, in order of likelihood:

• Prices stay flat or drift down. The efficiency gains are already in the numbers; there's no step-change to pass through, just continued pressure.
• Off-peak windows get more attractive. A framework optimized for sustained high throughput is exactly what makes discounted off-peak tokens profitable to sell. If you batch non-urgent work, see our cheapest DeepSeek API comparison.
• Third-party providers may tighten the gap. OpenRouter-class resellers running open weights don't have DeepInfer. Some of the price difference you see across providers is exactly this framework.

Should you deploy it?

Straight answer: almost certainly not — and I say that as someone sympathetic to the release. If you have to ask whether you can afford 64 SM100 GPUs for a decode cluster, you're not the audience. The realistic audience is: hyperscalers and large inference providers with Blackwell or Ascend fleets; research groups studying production serving (the code is the best-documented example of frontier-lab serving practice ever published); and teams building on the Chinese NPU stack who now have a reference implementation from the most demanding customer Ascend has.

For self-hosting V4.1 Flash on realistic hardware, the existing guides stand: our DeepSeek 4.1 Flash deployment guide covers the paths that actually work on accessible GPUs, and nothing in DeepInfer changes them — the framework doesn't even target that hardware.

Dated caveat: figures verified October 25, 2026 — GitHub repo stats (stars, creation date, license, hardware list) pulled live from the GitHub API and README; cluster throughput and per-GPU-hour revenue are from DeepSeek's technical report as reported by Yicai, original report text not independently reviewed; margin percentages are third-party estimates, not official figures. Accuracy is claimed only as of this date.