Serving topology

A release's topology decides how its GPUs are used. The KV cache and the router decide how requests use them.

Two phases

Generating a response has two phases:

• Prefill reads the whole prompt and builds the KV cache for it. It is compute-heavy and runs once per request.
• Decode produces output tokens one step at a time, reading the KV cache each step. It is memory-bandwidth-heavy and runs for as long as the output.

The two want different parallelism and batch sizes.

Aggregated

Each replica does both phases on the same GPUs.

request -> router -> replica (prefill + decode) -> tokens

Simple and efficient for models that fit on one server, and for short prompts. Most recipes start here.

Disaggregated

Prefill and decode run on separate pools. A serving unit pairs a prefill pool with a decode pool; prefill builds the KV cache and hands it to decode.

request -> router -> prefill pool --(KV transfer)--> decode pool -> tokens

Each pool is sized and parallelized for its phase. This suits large mixture-of-experts models, long prompts, and loads where prefill bursts would otherwise slow decode for everyone. DeepInfer's published benchmarks used 4 GPUs for prefill and 64 for decode.

Configure it under topology in the release. See Disaggregate prefill and decode.

Scale-up domains

KV transfer and expert parallelism move a lot of data between GPUs. Inside a scale-up domain (an 8-GPU NVLink server, or a 72-GPU rack) that traffic uses the fast fabric. Across domains it uses the network.

topology.placement: same-domain keeps every serving unit inside one domain. denv checks a domain has room for the whole unit before a deployment starts, and never splits a unit across domains. any allows crossing, for smaller models or when you accept the cost. See Hardware.

Expert and data parallelism

Mixture-of-experts models spread experts across GPUs (expert parallelism) and often run attention as several data-parallel ranks. The engine handles both inside a unit; recipes set the degrees. Data-parallel ranks need load balancing between them, which the router does, along with balancing across units.

KV cache

The KV cache holds the attention state of processed tokens. denv keeps it in tiers:

Tier	Holds
gpu	What the engine is using now.
host-memory	Recently used prefixes, evicted from GPU memory.
nvme	Larger, older prefixes on local disk.

With share: stage, a prefix computed on one replica can be reused by others in the stage. Long system prompts, few-shot examples and agent conversations that resend history all benefit. See Configure the KV cache.

KV-aware routing

The router keeps an index of which replicas hold which prefixes. For each request it prefers a replica that already holds most of the prompt, within the stage's routing policy and its load limits. Cached tokens are metered as cached_input and can be priced lower.

How a request flows

1. The router authenticates the key, checks limits, and converts the request through the API adapter.
2. It picks a fleet by the stage's routing policy, then a serving unit by prefix locality and load.
3. Prefill builds the KV cache for the uncached part of the prompt.
4. In a disaggregated unit, the KV cache moves to the decode pool.
5. Decode streams tokens; the adapter converts them into the API's response format.
6. Metering records the tokens for the customer.

When something fails

• A prefill pool member dies. In-flight prefills on it fail and are retried on another unit's prefill pool by the router. The unit is marked unhealthy until denvd restarts the engine and checks pass.
• A decode unit dies. Its in-flight streams end with an error the client can retry; new requests go elsewhere. The orchestrator replaces the unit.
• A host leaves a domain. Units that used it are taken out of rotation; the provisioner replaces the host or the unit moves to a domain with room.
• A spot host is reclaimed. Its units leave rotation; in-flight requests finish or fail over; the provisioner replaces the host within the fleet's max.
• A provider fails. Its fleets leave rotation, and the router sends traffic to the stage's other fleets, within residency.

On any failover, KV cache entries held only by the failed replica are lost; the next request for that prefix computes it again. Under priority routing, the base fleet scales up first and burst fleets scale down first.