Docs · how to

Route across fleets

Serve one stage from several fleets and providers, with a reserved base, burst capacity on different hardware, failover and residency.

A stage can serve from several fleets. The router decides which fleet gets each request by the stage's routing policy, and within that choice prefers the replica that already holds the request's prefix in its KV cache.

Policies

Policy What it does Use it for
priority Fill fleets in order; spill to the next when the current one is saturated A reserved or owned base with burst behind it
weighted Fixed shares, weights: { <fleet>: N } Moving traffic between providers gradually
latency The fleet answering fastest now Default
cost Lowest cost per million tokens that still meets the stage's alarms Price-sensitive traffic

KV cache locality applies inside every policy: among the replicas the policy allows, the router picks the one with the longest cached prefix, unless it is overloaded. See Serving topology.

Reserved GB200 base, H200 burst on another provider

The same weights can run on different hardware with a different recipe per fleet:

recipes:
  - name: ds-flash-h200
    version: 1
    extends: deepseek-v4.1-flash/sglang@2
    hardware: [h200]

release:
  model: ds-flash
  weights: hf://deepseek-ai/DeepSeek-V4.1-Flash-1017@c41a7e2
  recipe: deepseek-v4.1-flash/deepinfer@3
  topology:
    mode: disaggregated
    prefill: { gpus: 4, replicas: 2 }
    decode: { gpus: 64, replicas: 1 }

environment:
  name: ds-flash
  stages:
    - name: prod-us
      fleets: [gb200-rack, h200-burst]
      recipes: { h200-burst: ds-flash-h200@1 }
      replicas: { min: 2, max: 6 }
      routing: { policy: priority, order: [gb200-rack, h200-burst] }

gb200-rack serves with the release's recipe and topology. h200-burst serves the same weights with ds-flash-h200@1. Both are checked by the same golden prompts and parity suite, so a request gets the same model wherever it lands.

Autoscaling

The autoscaler adds and removes replicas within the stage's replicas range, from queue depth and tokens per second. Hosts are added within each fleet's own min and max. With priority, the base fleet scales first; the burst fleet grows only when the base is at its max or saturated, and shrinks first.

denv fleet scale h200-burst --min 0 --max 4

Failover

When a fleet's hosts fail health checks or its provider is unreachable:

  1. Its replicas leave the rotation.
  2. Traffic moves to the stage's other fleets under the same policy.
  3. The provisioner replaces failed hosts within the fleet's bounds.
  4. Replicas rejoin after they pass warmup and checks.

Requests already streaming from a failed replica end with an error; clients retry. KV cache entries on that replica are lost, so the first requests after failover recompute their prefixes.

Residency

    - name: prod-eu
      fleets: [eu-h200, eu-own]
      residency: eu

denv refuses fleets outside the residency, and the router never fails over outside it, even if every EU fleet is down.

Watch it

denv endpoint get ds-flash/prod-us
denv metrics ds-flash/prod-us

Both show each fleet's share of traffic, KV cache hit rate, latency, and cost per GPU-hour.

Next