Why every deployment prefetches, switches in batches, checks, compares against a reference, bakes, and rolls back on its own.

Every change to what a stage serves — new weights, a new engine version, one changed argument — is a new release, and reaches traffic only through a deployment. Each step below exists to stop a specific failure.

The steps

Step	Stops
Prefetch weights and engine images onto every target host while the old release serves	Long switches; capacity missing for the length of a download
Batches of serving units	A bad release reaching all traffic at once
Minimum healthy capacity at every moment	Customers seeing errors because too many units are switching
Warmup requests before checks	First customer requests paying for cold caches and graph compilation
Health checks and golden prompts	Units with wrong weights, broken templates or failed adapter loads taking traffic
Parity gate after each batch's checks, before bake	A release that runs but answers differently from the reference
Bake under alarms	Problems that only show under real load: latency, errors, throughput
Automatic rollback	Waiting for a person to notice

Details and defaults are in Rollout policy.

Serving units, not GPUs

For a disaggregated release, the unit of change is a serving unit: its prefill and decode pools together. Prefill and decode of one unit always run the same release, so KV cache formats and engine versions never mix inside a unit.

Engine upgrades are releases

There is no in-place engine upgrade. A new engine version enters through a new recipe version, which makes a new release, which goes through the steps above in every stage. The parity gate is where an engine regression is most likely to show.

Release speed across stages

Stages are ordered from small and fast to large and careful:

A canary stage with one unit, short bake, automatic promotion.
Production stages with larger batches only where capacity allows, longer bake, approval, and deploy windows.

A problem found in the canary never reaches production. A problem found in one production region does not reach the next. See Set up a pipeline.

One deployment per stage

Only one deployment runs on a stage at a time. A second waits. Pins (denv stage pin) refuse new deployments during incidents.

Rollback

Rollback deploys the previous release with the same batches and healthy floor, at full speed, without bake. It reuses the previous weights from local disk. A parity-fail during bake also rolls back, and the rolled-back release cannot be promoted. See Roll back.

A failing deployment

ds-flash@v9 changes the engine version in the recipe. It is deployed to prod-us: 4 units, batch 1, min-healthy: 75%, bake 2h, parity gate on.

10:02  prefetch      v9 weights already cached; engine image pulled on 4 units
10:09  unit u1       drain -> activate -> warmup 64 -> checks: health ok, golden 24/24
10:21  unit u1       in service; parity run 1: tool-calls 99.6%, exact 97.1%   pass
10:22  unit u2       drain -> activate -> warmup -> checks ok
10:34  unit u2       in service; parity run 2: tool-calls 93.8%                 FAIL
10:34  alarm         parity-fail (threshold 1)
10:34  rollback      u2 -> v8, u1 -> v8 (batch 1, min-healthy 75%)
10:51  rollback      complete; prod-us on v8; v9 marked not promotable

Two of four units ever served v9, never more than one batch at a time below full capacity. The report lists the failing parity cases, which point at tool-call parsing in the new engine version.

Everything is recorded

Each deployment keeps its plan, every command and result per host, alarm values, parity reports, approvals, and who started it. denv deployment list and denv deployment watch show them.

Deployment safety