deepinfer-scope

deepinfer/README.md · a679bb8 · L3–52 · 固定 commit 上游

这是固定版本源码的只读摘录；可在线阅读，也可通过上游链接查看原文件。

3DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.
4
5Supported models:
6
7- DeepSeek-V4.1-Flash
8
9Supported features:
10
11- Optimized kernels, model and sampler runtime
12- Parallelism: Attention DP / balanced CP, MoE EP + EPLB
13- DSpark and confidence-scheduled verification
14- SWA bounded replay
15- Asynchronous scheduling (zero-overhead batch scheduler)
16- Batch-invariant inference
17- Deterministic sampling
18- Structured outputs
19
20<details>
21<summary>Feature details and configuration</summary>
22
23- **Optimized kernels:** [FlashMLA](https://github.com/deepseek-ai/FlashMLA), [DeepGEMM](https://github.com/deepseek-ai/DeepGEMM), [DeepEP](https://github.com/deepseek-ai/DeepEP), [TileKernels](https://github.com/deepseek-ai/TileKernels), [DeepSelect](https://github.com/deepseek-ai/DeepSelect), and custom [TileLang kernels](deepinfer/model_executor/layers/tilelang_ops).
24- **DSpark and confidence-scheduled verification:** accelerate decoding with draft tokens and use confidence estimates to schedule verification. Set `DEEPINFER_ENABLE_MTP_VARLEN=1`; `DEEPINFER_USE_NEXTN_DECODE` controls the maximum draft length (default: 5).
25- **Decoder SWA bounded replay:** reduce decoder prefill work by replaying only a bounded suffix. Set `DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact` and `DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140` for the default V4.1 settings; the retained suffix must cover the SWA window plus the maximum backtracking length.
26- **Asynchronous scheduling:** overlap CPU batch scheduling with device execution. Enabled by default.
27- **Batch-invariant inference:** keep model results identical across different batch compositions. Enable with `DEEPINFER_BATCH_INVARIANT=1` (NVIDIA only).
28- **Deterministic sampling:** produce identical outputs for identical input tokens and sampling parameters. Set `DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0` and pass a fixed `seq_seed` to `SamplingParams` (NVIDIA only).
29- **Structured outputs:** constrain generation to a regex, JSON schema, or EBNF grammar. Pass `so_args={"schema_type": ..., "schema": ...}` to `SamplingParams`; see [examples](examples/_structured_output.py).
30
31</details>
32
33Supported hardware:
34
35- NVIDIA SM100 GPUs
36- Huawei Ascend 950 NPUs
37
38Required for a complete inference service (not included in this release):
39
40- A KV cache system
41- VE/Prefill/Decode disaggregation (most required engine functionality is already implemented)
42- Load balancing across data-parallel replicas for both prefill and decode
43
44## Getting started
45
46### Prerequisites
47
48- Linux
49- GCC >= 13.0
50- Python 3.12
51- CUDA 13 (NVIDIA only)
52- CANN 9.2 (Ascend only)