How to run LLM inference

People start with a platform question. Kubernetes or Slurm? EKS or GKE? An API or a GPU in the office? Those are late questions. Running inference is a chain of decisions: which model, what latency, where the GPUs come from, which copy serves this request, and what happens when that copy dies.

This is the short version of a guide I published on X: How to Run LLM Inference. The long version has the historical asides and the full layer model. This one is the path, and it links to the denv pages that implement a piece of it. denv does not make every decision on the ladder. It is one rung, and it is early.

A few terms

Inference is running a trained model. Training builds it. For most companies the ongoing bill is inference, because it runs on every request.

A token is the unit the model reads and writes, roughly three-quarters of an English word. Speed and price are both in tokens.

Prefill reads the prompt and builds state. It is usually limited by compute. Decode writes the reply one token at a time. It is usually limited by how fast the GPU can read memory. The 50.8 tok/s ceiling is that second limit, measured.

The KV cache is the state prefill stored, reused so decode does not reread the prompt. It lives in GPU memory and it grows with context and with concurrency. It often runs out before compute does.

The control plane decides how many replicas exist and where. The data plane does the request. denv is opinionated about both: the cluster is the control plane, the recipe’s engine is the data plane.

Five questions before any tool

1. Closed model behind an API, or an open-weight model you can download? Llama, Qwen, DeepSeek and the rest of the recipes are the second kind.
2. Steady traffic, spikes, or batch? Peak requests and prompt length matter more than the logo on the GPU.
3. Does the first token have to be fast, or does the job only have to finish by morning?
4. Can the prompts leave your account? If not, a shared API is out.
5. Who operates it? Self-hosting is cheaper only if someone owns it.

Memory math

Weights are about parameters times bytes per parameter. A 70B model at 16-bit is about 140 GB of weights. At 8-bit, about 70 GB. At 4-bit, about 35 GB. Then add KV cache, which can match or exceed the weights, and some runtime headroom. That tells you whether you are on one GPU, several in one server, or a smaller quant. Test quality on your prompts. A quant that wins a benchmark and fails your task is a failed quant.

The Llama 3.1 70B page uses this arithmetic and then refuses to invent a GPU count. That is deliberate.

The ladder

Climb it only when the rung you are on hurts.

1. Hosted closed APIs. Fastest way to learn what the product needs. You give up the model and you pay per token.
2. Hosted open-model APIs. Together, Fireworks, and the hyperscaler catalogs. Open weights, still no GPU to run. See Together and Fireworks.
3. Someone else’s dedicated endpoint. Baseten, Modal, Replicate and similar. More isolation, and you start paying for GPU time whether or not you fill it. See Baseten and Modal.
4. Your engine on GPUs you rent. Hyperscalers, neoclouds such as Lambda and Nebius, and marketplaces such as RunPod. This is where denv sits: you bring the account, denv runs vLLM or SGLang from a recipe.
5. Hardware you own or colocate, including Hetzner. Lowest cost per token only at steady volume, and only with a team that can take care of machines.

A practical order: prototype on an API, measure tokens and latency, move the steady part onto GPUs you run, keep the API as overflow. Do not build a platform first.

Engines and the layers above them

vLLM is the widely used open engine and the one the H100 numbers on this site were measured with. SGLang is the other one denv will pin in a recipe. TensorRT-LLM is NVIDIA’s stack. llama.cpp and Ollama are for a laptop, not for this product.

Above the engine, add Kubernetes when you have more than a handful of replicas. Add llm-d when prefix caching and load should pick the replica. Add a fleet layer when you span clouds. denv is that fleet layer in the specific shape of one cluster, not a second Kubernetes per cloud. SkyPilot is the launcher you would use if you wanted to own the rest yourself.

Production is repair, not a dashboard

Demand, cost and security are in the long guide. The part denv is built around is failure. Health checks, a disruption budget, failover, and a repair ladder that includes the GPU driver. A GPU that falls off the bus should not be a new project.

Cost per million tokens is real, and it is mostly utilization. A dedicated H100 at one request at a time can lose to a token API even when it is faster. The 213 tok/s post does that division for one rental, $4.50 an hour, and then points at the 1,817 tok/s aggregate as the number that matters. denv does not sell that hour.

If you want the full guide, read it on X. If you want the implementation, start at run inference or write to anup@denv.ai. The product is early. Job-shaped pages, which refuse to invent a latency, are voice, coding agents, document AI, computer use, and video.