One Kubernetes cluster, many GPU clouds

Most teams serving models do not have a GPU shortage so much as a geography problem. Capacity sits in a hyperscaler, a neocloud, an aggregator, and a rack in someone’s own building. Each has a price, a queue, and a different API. A serious setup ends up in three to five places.

The default architecture is a Kubernetes cluster per provider, plus something to federate them. The hard problems move into the glue: networks, cross-cluster scheduling, failover, and who fixes a node at 3am.

The other shape is one Kubernetes cluster whose nodes live at many providers. For training that is usually a mistake. For inference it is the simpler design, if you follow a few rules. This is that argument, shortened from the X article. The product page is multi-cloud GPU inference.

Replicas do not talk to each other

Training wants every GPU on one fabric. An inference replica is one model on the GPUs it needs, often one machine, sometimes a few in the same rack. The request lands on one replica and finishes there. Replicas do not talk to each other.

So you keep each replica inside one site, and you let different replicas live anywhere. What has to cross a site is control traffic, the gateway’s request on the way in, and object storage. The token loop stays local.

Three rules

1. Nodes join outbound. The machine dials the control plane. No inbound ports, no VPN project with the provider’s network team. This is why a colo box can join the same way a cloud VM does.
2. A mesh carries control traffic and the occasional cross-site call. It is not the data plane. Do not stretch one flat pod network across the internet and hope.
3. Pods of one workload do not straddle sites. A service prefers endpoints in the same site.

A site is a label, like a zone: nebius-eu, lambda-us, own-dc. On denv that label is a site with a GPU type and, for a cloud account, a minimum and a maximum.

Routing and failover

Spraying requests across every replica wastes the prefix cache. The replica that served the last turn can skip most of the prefill. Send the next turn to another cloud and you recompute it.

Pick the site first: health, load, latency, and stickiness. Fail over only when that site cannot answer. Inside the site, llm-d picks the replica that likely already holds the prefix, unless that replica is overloaded.

Retry only a request that produced no tokens. Once tokens have streamed, a silent retry elsewhere is a different answer.

Repair is the same ladder everywhere: detect, contain, recover, reset or reboot or reinstall the driver, then prove it. A node returns after a smoke test, not after a timer. If many nodes at one site fail together, evacuate the site. That is a provider incident.

“No capacity”, “over quota” and “bad credentials” are different errors. The first can try the next provider. The second needs a person. The third means stop touching that account.

What you give up

Anything that assumes one cloud. A load balancer that only sees one provider’s nodes. A disk that cannot follow a pod to another provider. A cluster autoscaler written for three hyperscalers. You replace those once, in the control plane, instead of once per provider.

Do not do this if all of your GPUs are already in one cloud and you have a platform team. Do not do this for large training. Do not split one replica across providers. If data cannot leave a region, the site is a compliance boundary before it is a capacity pool. And someone has to run the one control plane. Hosted denv is that someone, or you self-host it.

Adding a provider should be adding a site. Losing a provider for an hour should be a failover, not a new design. That is the shift. GPU clouds become providers on denv.