Architecture

denv keeps a record of what should run where, turns changes to that record into commands for individual hosts, runs them safely, and serves traffic to the result.

                         denv control plane (multi-tenant)
  ┌───────────────────────────────────────────────────────────────────┐
  │                                                                   │
  │  API server ---> Orchestrator ---> Command service <---------┐    │
  │  (records)       (deployments,     (per-host queues)         │    │
  │      │            rollbacks)                                 │    │
  │      v               ^                                       │    │
  │  Provisioner     Metrics <--- Metering <--- Parity runner    │    │
  │  (fleet sizes)   (alarms)     (usage)       (reports)        │    │
  │                                                              │    │
  │  Recipe and test service      Artifact store (engine images) │    │
  └──────┬───────────────────────────────────────────────────────┼────┘
         │ launch / terminate                                    │ outbound mTLS
         v                                                       │
  ┌──────────────────────── your providers ──────────────────────┼────┐
  │                                                              │    │
  │  Hosts: denvd + engine (prefill and decode pools) -----------┘    │
  │     ^                                                             │
  │     │ requests, KV locality      Regional weight cache <-- your   │
  │     │                                                    storage  │
  │  Router / gateway (endpoints, keys, limits, KV index)             │
  └───────────────────────────────────────────────────────────────────┘
         ^
         │ customers' requests

Components

API server. Stores every object in Concepts: providers, fleets, hosts, engines, recipes, releases, environments, deployments, customers, keys, prices. It validates, checks permissions and records intent. It does not act.

Orchestrator. Turns a deployment into a plan: which serving units, in which batches, which commands on which hosts. It enforces the stage's rollout policy, runs the parity gate, watches alarms and rolls back. Every step is persisted; a restart resumes where it stopped.

Command service. One queue per host. Each denvd keeps an outbound connection, takes its next command, runs it and returns the result. A host only sees its own queue.

Provisioner. Keeps each fleet between min and max through the provider's API, launching from images with denvd. Replaces failed and reclaimed hosts. Never touches owned hosts.

Recipe and test service. Runs a recipe's tests on its hardware before it is published, and records results.

Artifact store and weight caches. Engine images by digest. Weights stay in your storage; a regional cache near each fleet pulls them once, verifies the digest, and serves the hosts in that region.

Router / gateway. Serves endpoints. Checks keys and limits, applies the recipe's API adapter, and balances across data-parallel replicas and prefill and decode pools. It keeps an index of which replica holds which prefixes in its KV cache and sends requests there. It can run in denv's cloud or in your account.

Metering. Counts input, cached-input, output and reasoning tokens per request, per customer, from the router. Feeds usage, exports and webhooks.

Parity runner. Sends parity suites to a stage and to its reference, compares, and writes reports.

Metrics. Engine and host metrics from denvd, router metrics, metering. Feeds alarms, autoscaling and dashboards.

Why not Kubernetes

Kubernetes is a good fit for many small, interchangeable containers in one cluster on one flat network. An inference service is different:

• Few, large, long-lived units. A serving unit can be 68 GPUs in one rack. The hard part is not packing; it is changing such units safely.
• Topology is the constraint. Where GPUs sit in a scale-up domain matters more than which node has free CPU.
• Many providers, no shared network. A stage may use your racks, a GPU cloud and a hyperscaler. One cluster spanning them is fragile; one cluster per provider multiplies operations.
• Versioned change is the product. Weights, engine, recipe and config ship as one release through stages, with parity and rollback.

So denv is host-centric: an agent on every host pulls commands; a control plane holds environments, stages and releases; an orchestrator controls release speed. The model follows the deployment systems large cloud providers built for their own services. Werner Vogels described one in "The Story of Apollo - Amazon's Deployment Engine" (allthingsdistributed.com, 2014), the inspiration for this design.

Kubernetes users can still use denv: hosts in a Kubernetes cluster can run denvd like any other machine.

What runs where

Runs in denv's cloud	Runs on your hosts and storage
API server, orchestrator, command service, provisioner, metering, parity runner, artifact store	Weights, engines, KV cache, prompts and outputs in memory, logs
Router, unless you run it in your account	Router, if you choose

Prompts and outputs pass through the router to your hosts. They are not stored unless you turn on request logging. See Security.