Releasing the inference engine behind its own service
China's DeepSeek on Oct. 20 open-sourced "DeepInfer," the framework it uses for inference, the process of producing answers with a trained model, in its own services. It announced the release on its official X account the same day (same day Japan time). The license is Apache 2.0, which permits commercial use, and the project drew more than 20,000 stars on GitHub shortly after release.
DeepSeek said DeepInfer supports large-scale serving of "DeepSeek-V4.1-Flash," its flagship small model. That is the only supported model so far, and by default it uses "V4.1-Flash-1017," the improved version released on Oct. 17. The company said it will also release inference code and improvements for future models through this project.
Runs on both NVIDIA and Huawei
DeepInfer supports both NVIDIA's latest-generation GPUs (the SM100 generation) and Huawei's "Ascend 950." It includes mechanisms for splitting processing across multiple chips, "speculative decoding," which speeds up the whole process by having a small model predict answers first, and output in fixed formats such as JSON. The company said the design is based mainly on "vLLM," a widely used inference engine, and also drew on "SGLang" and NVIDIA's "TensorRT-LLM."
Some features, such as returning the same answer every time for the same input, work only on NVIDIA GPUs. Lian Jye Su, an analyst at British research firm Omdia, said that because the framework supports Huawei's chips while continuing to depend on NVIDIA, Huawei still lags NVIDIA in some areas.
Not enough for production serving on its own
What was released is the "offline" inference component, which processes requests in batches. It does not include what production services require: a mechanism for reusing intermediate computation results (KV cache), a mechanism for splitting input processing and output generation across separate machines, or load balancing across multiple machines. DeepSeek also explained that it is not a complete online service.
Performance was shown only as a graph from measurements using 4 GPUs for input processing and 64 for generation, with no specific figures or comparisons with other software. There has been no outside verification yet.
The same week, the United States barred the use of DeepSeek at federal agencies. China has pushed back, saying the move turns technology into a political issue.
