Docs · explanation

From engine to service

What an open inference engine gives you, what running it as a production service needs on top, and where denv fits.

An inference engine takes a model and GPUs and turns requests into tokens quickly. Running it as a service your applications depend on, or that you sell to others, needs much more around it. denv is that layer.

What labs now publish

Model labs increasingly publish the engine they run in production, not only the weights. In October 2026 DeepSeek released DeepInfer, the engine behind its DeepSeek-V4.1-Flash API, under Apache 2.0, together with deepseek-recipe, the library that makes requests and responses match the official API.

DeepInfer's README is explicit about the gap. It lists, as required for a complete inference service but not included in the release:

  • a KV cache system;
  • prefill/decode disaggregation (with most of the engine functionality already implemented);
  • load balancing across data-parallel replicas, for prefill and for decode.

That is not a shortcoming of the release. It is where an engine ends and a service begins, and it is the part every operator otherwise builds alone.

Engine and service, side by side

Concern The engine The service also needs
Model execution Kernels, batching, parallelism inside a replica —
Placement Runs where you start it Units placed inside one scale-up domain; enough free GPUs before switching
Prefill and decode Can run either role Separate pools, sized separately, paired into serving units, KV moved between them
KV cache GPU memory of one replica Tiers (GPU, host memory, NVMe), prefixes shared across replicas
Routing One process Balancing across data-parallel replicas and pools, sending requests where their prefix is cached
API Often an OpenAI-compatible server Chat Completions, Responses and Messages that behave like the model's official API, including tool calls and reasoning
Access None or one key Keys and rate limits per team or application, usage, and for token sellers prices and exports
Correctness Whatever it computes Golden prompts and parity against a reference, before and after every change
Change Restart with new flags Versioned releases, staged rollouts, automatic rollback
Capacity The machine it is on Fleets across providers, reserved first, spot for bursts, failover when a provider fails
Operations Engine metrics Per-fleet cost per GPU-hour and per million tokens, latency, cache hit rate, alerts

What denv does with each

Why recipes

The combinations multiply: several engines, each with versions; several hardware families; aggregated or disaggregated; different parallelism per model. Most of the work of serving a new model is finding a combination that is fast, cheap and correct. A recipe records one such combination; benchmarks measure its speed, throughput and cost per workload, so you can choose one instead of searching yourself.

Where open-source pieces fit

There are good open-source projects for parts of this: disaggregated serving frameworks, KV cache stores and transfer libraries, routers for data-parallel engines, and production stacks for single engines. Most assume one engine, or one Kubernetes cluster on one network.

denv differs in four ways:

  • Across providers. One stage can serve from your racks, a GPU cloud and a hyperscaler at once, with failover between them.
  • Topology-aware. Placement understands 8-GPU servers and 72-GPU racks as units.
  • Engine-agnostic. vLLM, SGLang, TensorRT-LLM and DeepInfer behind the same router, KV cache, API and metering.
  • Not tied to a cluster. Hosts run an agent and are managed directly. See Architecture.

Where an engine or library already does something well, denv uses it rather than replacing it.