denvd sidecar

denvd is the one process denv runs on a customer's machine. It is a static binary with four verbs. It opens one outbound WebSocket to the control plane and never listens on a port. It runs environments with Docker Compose, keeps their datasets on disk (or in a ZFS pool), and executes commands idempotently.

Verbs

denvd join    --server URL --token T [--state FILE] [--data DIR] [--zfs-pool POOL] [--env-file FILE]
denvd run     [--server URL] [--state FILE] [--data DIR] [--zfs-pool POOL]
denvd install --server URL --token T [--data DIR] [--zfs-pool POOL] [--env-file FILE]
denvd version

Verb	What it does
join	Runs in the foreground. On first connect it presents the join token; the control plane assigns a host id and a host secret, which are saved to the state file. Later connects use the saved identity.
run	Same loop as join but refuses to start without a saved identity; a token is not required.
install	Linux, root only. Copies itself to /usr/local/bin/denvd, writes /etc/denv/denvd.env (mode 0600) with DENV_SERVER, DENV_JOIN_TOKEN, DENV_DATA_DIR, DENV_ZFS_POOL, writes /etc/systemd/system/denvd.service, then runs systemctl daemon-reload and systemctl enable --now denvd.
version	Prints the build version (set at build time; dev otherwise).

Flags and environment

Every flag falls back to an environment variable.

Flag	Env	Default	Meaning
--server	DENV_SERVER	— (required)	Control plane base URL, http:// or https://. The stream URL is derived: wss://<host>/v1/hosts/stream.
--token	DENV_JOIN_TOKEN	—	Join token (djt_…). Required on the first join unless run is used. Never persisted.
--state	—	/etc/denv/host.json as root, else ~/.denv/host.json	Identity file: server, host_id, secret, data_dir, zfs_pool. Mode 0600.
--data	DENV_DATA_DIR	/var/lib/denv as root, else ~/.denv/data	Data directory.
--zfs-pool	DENV_ZFS_POOL	—	ZFS pool name. Storage becomes zfs only when the pool exists (zpool list <pool> succeeds).
--env-file	—	—	Reads KEY=VALUE lines into the environment before the flags above are resolved (used by cloud-init).

Saved state wins for server, data_dir, and zfs_pool unless the flag is given.

What the sidecar does

Connect and identify

On each session the sidecar sends a hello frame with its host id, its secret (or the join token when it has no id), and the machine's capabilities: hostname, OS, arch, CPU count, memory, free disk under the data dir, storage backend (zfs or dir), Docker server version, whether docker compose exists, whether network namespaces are usable (Linux as root), and the denvd version. A heartbeat frame is sent every 10 s. Reconnects back off from 1 s to 30 s. A PolicyViolation close from the control plane (bad token, unknown host, rejected secret) is fatal and the process exits.

Execute commands

Each command carries a command id. Results are written to <data>/commands/<id>.json before being sent; a command id seen again is answered from that file without re-executing. One command runs at a time per environment.

Command kinds and what they do on the host are listed in Protocol. In short: env.create renders a compose file and starts the stack, env.snapshot freezes containers and captures every volume, env.fork clones a snapshot into a new environment, env.rollback restores a snapshot, env.exec runs docker compose exec, env.put/env.get move tar archives with docker cp, verify.collect/verify.run implement grading, env.destroy stops the stack and removes its datasets.

Data directory layout

Path	Contents
<data>/envs/<env id>/compose.yaml	The rendered compose file for the environment.
<data>/envs/<env id>/spec.json	Services, volumes, egress, saved so rollback can re-render after a restart.
<data>/envs/<env id>/vol/<volume>/	Volume data (dir storage).
<data>/snaps/<env id>/<snapshot>/	Snapshot copies (dir storage).
<data>/commands/<command id>.json	The stored result of every command (idempotency log).
<data>/verify/<deployment id>/submission/	Root-owned copy of the collected paths for one verify.
<data>/verify/<deployment id>/logs/	The grader's /logs/verifier mount.

With ZFS the volumes are datasets <pool>/envs/<env id>/vol/<volume> mounted at /<pool>/envs/<env id>/vol/<volume>, snapshots are ZFS snapshots named after the version, and forks are clones.

Storage backends

Backend	Reported as	Snapshot	Fork	Rollback
ZFS pool	zfs	zfs snapshot per volume dataset	zfs clone	zfs rollback
Directory	dir	Recursive copy of vol/ into snaps/	Recursive copy out of snaps/	Recursive copy over vol/

dir is correct and slow; it is the backend on a laptop and on any Linux host without a pool.

Compose project naming

The Docker Compose project name is the environment id in lower case. The stack's network is <project>_default.

What the sidecar refuses

The compose subset is enforced by the control plane before a spec reaches the sidecar (see Compose). On the host, the sidecar additionally refuses, during verify, any collected path that is a symlink, FIFO, device, or socket, and trees that exceed 256 MiB, 20,000 files, or 32 directory levels.

Install script

curl -fsSL https://get.denv.ai | sh -s -- --token <join token> [--server URL] [--zfs-pool POOL] [--data DIR]

The script (scripts/install.sh in the backend repository) requires Linux, root, and Docker on the host. It detects amd64 or arm64, downloads <releases>/denvd-linux-<arch> with curl or wget, installs it to /usr/local/bin/denvd (DENV_BIN_DIR overrides), and runs denvd install --server … [--token …] [--zfs-pool …] [--data …].

Flag	Env	Default
--server	DENV_SERVER	https://api.denv.ai
--releases	DENV_RELEASES	https://get.denv.ai
--token	DENV_JOIN_TOKEN	—
--zfs-pool	DENV_ZFS_POOL	—
--data	DENV_DATA_DIR	—

The control plane also serves the binaries itself at GET /denvd (amd64) and GET /denvd-linux-{amd64|arm64} from its releases directory; the command field returned by POST /v1/join-tokens uses that route.

systemd unit

Written by denvd install:

[Unit]
Description=denv sidecar
After=network-online.target docker.service
Wants=network-online.target

[Service]
EnvironmentFile=/etc/denv/denvd.env
ExecStart=/usr/local/bin/denvd join --state /etc/denv/host.json
Restart=always
RestartSec=3
LimitNOFILE=1048576

[Install]
WantedBy=multi-user.target

The join token in /etc/denv/denvd.env is consumed on the first connect; after that the identity in /etc/denv/host.json is what authenticates the host, and the token line is ignored (a used token is rejected by the control plane, but join only presents it when no identity is saved).

Provisioned machines

When the control plane provisions a machine (Hetzner today), the cloud-init it passes installs ca-certificates curl docker.io docker-compose-v2 zfsutils-linux, writes /etc/denv/join.env with DENV_SERVER and DENV_JOIN_TOKEN, downloads denvd from the control plane's denvd URL, and runs denvd install --env-file /etc/denv/join.env. It does not create a ZFS pool; such a host reports storage: dir unless a pool is configured afterwards.