Reliability

Health checks, then auto-repair.

Detect a bad GPU, drain it, fail over, then reset, reboot, reinstall the driver, or replace the machine.

GPUs do not fail like CPUs. They fall off the bus. They retire memory pages. NVLink starts erroring. They throttle on heat. Sometimes they just get slow, and the node still looks fine.

denv runs the same ladder at every provider and on servers you own. The homepage walks it as five steps. This is the longer version.

Detect

Health checks read GPU signals: XID errors from the kernel, ECC, NVLink, temperature, and a smoke test. XID 79 is the one in the example log: GPU fell off the bus. Silent nodes are in the docs because a green node can still be a bad GPU.

A check becomes a node condition. Nothing waits for a person to notice a dashboard.

Contain

Cordon the node. Drain it inside the recipe’s disruption budget. The budget is there so auto-repair cannot take the last replica. If you run one replica, a drain is an outage. Run more than one, or allow another site, if the endpoint has to stay up. See failover.

Recover

Traffic moves to healthy replicas. Same provider or another one. The endpoint stays the same URL.

Repair

The ladder, in order, each step with a deadline:

  1. Reset the GPU.
  2. Reboot.
  3. Reinstall the GPU driver. This is the driver-management step. The join installs what the node needs; repair reinstalls the driver when reset and reboot are not enough.
  4. Replace the machine if it is a provider account denv can launch in. On servers you own, denv cannot buy you a new box. It opens a ticket and keeps the evidence.

Fatal faults skip ahead. A node does not return because a timer fired. It returns after the GPU smoke test passes.

Prove

Every step is recorded. You get a signed monthly uptime report per GPU and per provider. That is a record. The terms do not promise an uptime number.

A bad site is not twenty bad GPUs

When several nodes at one provider fail together, repairing them one by one is the wrong move. Evacuate the site and let failover carry the traffic. Treat “no capacity”, “over quota” and “bad credentials” as different problems when new machines are launched.

The same cluster is real Kubernetes. kubectl still works while this runs. Concepts names the objects.