llm-d, and why denv runs it for you.

llm-d is an open stack for serving models on Kubernetes, built around vLLM. It routes a request to the replica most likely to already hold that prompt’s prefix, and it can split prefill and decode onto different pools. The project’s own site and the vLLM docs are the spec. This page is the commercial wrapper.

What you would assemble yourself

A working llm-d install is the engine, the router, the Gateway, the Inference Extension, a cache, dashboards, and a policy for what happens when a GPU dies. Each piece moves. Pinning them so a deploy on Tuesday matches a deploy on Thursday is the actual work.

denv puts that pin in the recipe:

• simple: plain vLLM replicas.
• standard: llm-d with cache- and load-aware routing.
• large: prefill/decode split and wide expert parallelism for models that need it.

The image digest and the weights revision are part of the pin. A recipe is tested on its GPU type before it serves.

What denv adds that llm-d does not try to be

llm-d does not launch machines at Lambda, Nebius or AWS, and it does not join your Hetzner box. It does not walk a GPU reset, reboot and driver reinstall. It does not fail a site over. Those are the cluster’s job. denv is that cluster: one control plane across providers, like EKS on top of EC2, for inference.

You still get a kubeconfig. If you want to run llm-d yourself on Kubernetes you already operate, self-host denv or use the upstream project. Both are reasonable. denv is for teams that want the endpoint, the repair and the provider list without staffing the glue.

The public speed numbers on this site are the H100 runs, not an llm-d marketing claim. Read them in the benchmark and the throughput note.