Docs · how to

Roll back

What happens when an alarm or a parity failure rolls a deployment back, how to roll back by hand, and how to hold a stage on one release.

Rolling back means deploying the previous release. Because a release pins weights, recipe, engine image and every argument, the previous release is exactly what ran before.

Automatic rollback

A deployment rolls back on its own when, during the rollout or its bake:

Trigger Example
A replica fails warmup, health checks or golden prompts The engine starts but a golden prompt's tool call is missing
An alarm fires ttft-p99, itl-p99, error-rate, throughput-drop over threshold
The parity suite fails parity-fail: 1 and the suite misses its pass thresholds
A custom alarm fires Your own metric under alarms.custom

With rollout.on-alarm: pause, the deployment stops between batches instead and waits for you.

What happens, replica by replica

v8 deploying to prod-us, batch 1 of 4
  replica r1  prefetch ok  ->  drain  ->  switch  ->  warmup ok  ->  checks ok  ->  in service
  bake        parity-fail: tool-calls 96.2% (needs 99%)
rollback to v7
  replica r1  drain  ->  switch to v7 (weights still cached)  ->  warmup ok  ->  checks ok  ->  in service
  replicas r2-r4 never left v7
deployment  rolled_back
  • Only replicas already on the new release are switched back.
  • The previous release's weights are still on local disk, so switching back does not download them again.
  • Healthy capacity stays at or above min-healthy throughout.
  • Rollbacks skip bake.
  • A release that was rolled back cannot be promoted from earlier stages.

Roll back by hand

denv rollback ds-flash/prod-us

This deploys the release that ran before the current one, under the stage's rollout policy, at full speed. It needs the deployer role.

To go to a specific older release:

denv deploy ds-flash/prod-us --release ds-flash@v6

Hold a stage on one release

While you investigate, stop anything else from reaching the stage:

denv stage pin ds-flash/prod-us --release ds-flash@v7 --reason "parity drift on tool calls, see incident 112"

New deployments and promotions into the stage are refused until:

denv stage unpin ds-flash/prod-us

Find out why

denv deployment list ds-flash
denv deployment watch <deployment-id>
denv parity report ds-flash/prod-us
denv logs ds-flash/prod-us

Every step of a deployment is recorded with its result, including the alarm values and the parity report that triggered the rollback.

Next