Control plane ↔ sidecar protocol

Each denvd keeps one outbound WebSocket to GET /v1/hosts/stream on the control plane. Every message in either direction is one JSON frame. The control plane sends commands; the sidecar answers with acks, progress lines, results, and unsolicited events. Nothing on the control plane ever dials a host.

Frame

{
  "type": "command",            // hello | command | ack | progress | result | event | heartbeat
  "host_id": "host_…",
  "command_id": "cmd_…",        // commands, acks, progress, results
  "kind": "env.create",         // commands; events use it as the event kind
  "env_id": "env_…",            // commands addressed to an environment
  "payload": { … },             // kind-specific
  "ok": true,                   // results only
  "error": "…",                 // results, when ok is false
  "message": "…",               // progress lines
  "sent_at": "2026-10-12T00:00:00Z"
}

Session

1. The sidecar dials wss://<server>/v1/hosts/stream (ws:// for an http:// server). The read limit is 16 MiB per frame.
2. Its first frame is hello with payload {host_id, token, capabilities}. On the first ever connect host_id is empty and token is the join token (djt_…); afterwards host_id is set and token is the host secret (dhs_…).
3. The control plane admits the host. A new host consumes the join token (single use, 409-style rejection when used or expired), is assigned an id, and receives one hello frame back with {host_id, secret}, which it persists. A known host has its secret hash checked. Any refusal closes the socket with WebSocket status 1008 (policy violation); the sidecar treats that as fatal and exits.
4. The sidecar sends a heartbeat frame every 10 seconds; the control plane records last_seen_at on each. The grace window on the control plane is 60 seconds.
5. When the stream drops, the host is marked unreachable and the sidecar reconnects with backoff from 1 s to 30 s. Reconnecting as the same host replaces any older connection.

Capabilities

Sent in hello and returned by host.report:

Field	Meaning
hostname, os, arch, cpus, memory_mb, disk_gb	The machine. disk_gb is free space under the data dir.
storage	zfs when a pool is configured and present, else dir.
docker	Docker server version, "" if absent.
compose	Whether docker compose exists.
netns	Linux and root.
public_ip	Optional.
denvd_version	Build version.

Command lifecycle

A command is recorded in the control plane's commands table before it is written to the stream, so a crash between the two leaves a sent row rather than a silent loss. The sidecar replies ack when it has the frame, zero or more progress frames while working, then exactly one result with ok and either payload or error. The control plane records progress lines as events on subject cmd:<command id> and stores the result on the command row.

Idempotency. The sidecar writes every result to <data>/commands/<command id>.json before sending it. A command id it has already completed is answered from that file without executing again, so re-delivery after a reconnect is safe. One command runs at a time per environment on a host.

Timeouts. The control plane bounds each phase at 15 minutes (orchestrator CommandTimeout); env.exec uses the request's timeout_sec plus 10 s. A timed-out command is marked failed on the control plane; the sidecar may still finish it and store the result.

Command kinds

host.report

Payload {}. Result {capabilities, environments, load_avg} (the sidecar today fills capabilities only).

env.create

Payload:

{"version":"ver_…","services":[Service…],"volumes":["pg","home"],"from_snapshot":"","warm":["cmd","arg"],"egress":"deny"}

Prepares one dataset per volume, renders a compose file, runs docker compose up -d --wait --force-recreate --quiet-pull --remove-orphans, then runs the egress probe. Result: {endpoints: {service: "127.0.0.1:port"}, started_in, digests: {service: image@sha256}, storage, egress} where egress is allowed or denied (verified). The warm field is carried in the payload but the warm command itself is sent by the control plane as a separate env.exec in the warm phase.

Service (as rendered by the control plane):

Field	Meaning
name, image	Image is the reference from the compose file; digests are read back after up.
env, ports, volumes ("<volume>:<path>"), health ({test, interval, retries}), depends_on, command, user	As in the compose file.
resources	{memory_mb, pids, cpus}; zero means the sidecar default (4096 MiB, 2048, 2.0).

env.snapshot

Payload {name, keep_paused?: ["service"]}. Checkpoints databases (Postgres CHECKPOINT, Redis BGSAVE, MySQL FLUSH TABLES), pauses every container that is not already paused, captures every volume with the storage backend under name, then unpauses everything except keep_paused and waits for health (90 s). Result {snapshot, datasets, frozen_for, consistent: true}.

env.fork

Payload {from_env, from_snapshot, services, volumes, egress}. Addressed to the new environment id. Clones the snapshot's volumes into the new environment, renders, starts, probes egress. Result {forked_in, endpoints}.

env.rollback

Payload {snapshot}. Stops the stack, restores every volume from the snapshot, re-renders from the saved spec, starts. Result {snapshot, endpoints}.

env.exec

Payload {service, cmd: [...], timeout_sec?} (default 120 s). Runs docker compose exec -T <service> <cmd…>. Result {exit_code, stdout, stderr}; a non-zero exit is still ok: true.

env.put

Payload {service, path, tar_b64}. Creates path in the service and extracts the archive there with docker cp. Result {path}.

env.get

Payload {service, path}. Result {tar_b64}: a tar of path, limited to 32 MiB.

verify.collect

Payload {id, agent_service, collect: ["/abs/path"]}. Pauses the agent service, copies each path with docker cp -a=false (symlinks are not followed) into <data>/verify/<id>/submission/, then walks the copy with lstat. Result {files, bytes, missing: [...], refused: ["<path>: <reason>"]}. When anything is refused the whole copy is removed. The agent service stays paused.

verify.run

Addressed to the grader's fork. Payload {id, verifier: {image|service, command, collect, timeout_sec}, refused}. With refused: true returns {reward: 0, source: "refused"} without running anything. Otherwise runs the verifier (see Compose, verifier contract). Result {reward, exit_code, stdout, stderr, source, ran_for}; stdout/stderr are tails.

env.destroy

Payload {}. docker compose down and removal of the environment's datasets. Result {destroyed: "<env id>"}.

Declared, not implemented

egress.set, clock.set, ingress.set exist as kinds in the protocol package; neither the control plane sends them nor the sidecar handles them (the sidecar answers unknown command kind).

Events

Frames of type event from the sidecar are stored on subject host:<host id> with kind as the event kind. The sidecar does not emit any today.

Driver hosts

A host of kind driver:<provider> has no stream. The hub runs its commands in-process through the provider's driver, records them in the same commands table, and keeps the driver's per-environment state (provider sandbox id, snapshot id) in host_env_state. The payloads and results are the same shapes as above; a command the provider has no primitive for fails with <provider>: <kind>: not supported on <provider> yet or a more specific reason. See Targets.