Disaggregate prefill and decode

Prefill processes the prompt in one compute-heavy pass. Decode produces output one token at a time, limited by memory bandwidth. In an aggregated replica they share GPUs and slow each other down. A disaggregated release runs them in separate pools, each with its own parallelism, and moves the KV cache from prefill to decode over the domain's fabric.

When to use it

Use disaggregated	Stay aggregated
Large mixture-of-experts models that need many GPUs for decode	Models that fit on one server
Long prompts, such as agents with large contexts	Short prompts and short outputs
You need predictable inter-token latency under mixed load	You want the simplest setup
Your engine supports it (see Engines)	Your engine does not

What the engine gives you and what denv adds

DeepInfer's README says a KV cache system, prefill/decode disaggregation and load balancing across data-parallel replicas are required for a complete inference service but are not included in its release, while most of the engine functionality disaggregation needs is already implemented. Other engines vary.

denv supplies the operator side:

• Placing both pools of a serving unit inside one scale-up domain.
• Starting, health-checking and replacing each pool.
• Routing each request to a prefill replica, then to the decode replica that receives its KV cache.
• Sizing pools separately and scaling serving units within the stage's replicas.
• Rolling out a new release pool by pool, with checks.

See Engine vs service.

Example: DeepSeek-V4.1-Flash on GB200 NVL72

DeepInfer's published benchmark used 4 GPUs for prefill and 64 for decode, each within one scale-up domain. The catalog recipe follows that layout:

release:
  model: ds-flash
  weights: hf://deepseek-ai/DeepSeek-V4.1-Flash-1017@c41a7e2
  recipe: deepseek-v4.1-flash/deepinfer@3
  topology:
    mode: disaggregated
    prefill: { gpus: 4, replicas: 2 }
    decode: { gpus: 64, replicas: 1 }
    placement: same-domain

One serving unit here is two prefill replicas and one decode replica: 72 GPUs, which fills one NVL72 domain. replicas: 2 on the stage means two domains.

domain 1 (72 GPUs)                      domain 2 (72 GPUs)
  prefill 4 GPUs  --KV-->                 prefill 4 GPUs  --KV-->
  prefill 4 GPUs  --KV-->  decode 64      prefill 4 GPUs  --KV-->  decode 64

Sizing the pools

There is no universal ratio. Start from the recipe and adjust from what you measure:

• Prompts long, outputs short: prefill is the bottleneck. Add prefill replicas.
• Prompts short, outputs long: decode is the bottleneck. Keep prefill small.
• Many requests share a prefix: KV cache hits skip prefill work. Fewer prefill replicas may be enough. See Configure the KV cache.

Each change is a new release, so it goes through a canary like any other.

Placement

placement: same-domain (the default for disaggregated) keeps a serving unit's pools inside one domain, so the KV cache moves over NVLink rather than the network. The dry run refuses a stage whose fleets have no domain large enough:

denv release --dry-run

error  stage prod-us: fleet h200-use1 domain is 8 GPUs; serving unit needs 72 (same-domain)

Use placement: any only if your engine and network support KV transfer between domains and you have measured it.

Watch the two latencies

denv metrics ds-flash/prod-us

Metric	Controlled mostly by	If it rises
Time to first token (ttft-p99)	Prefill pool, queueing, KV cache hits	More prefill replicas, better prefix sharing
Inter-token latency (itl-p99)	Decode pool	More serving units, check decode batch size

Set alarms on both so a release that shifts the balance rolls back:

alarms: { ttft-p99: 1500ms, itl-p99: 60ms, error-rate: 1% }

See Serving topology and Rollout policy.