# denv — The environment company denv is the environment company. Environments you can version, fork per rollout, and grade apart from the agent: build one in Harbor, Inspect, OpenEnv, verifiers, HUD, Karotte, or your own framework, and denv runs it as a whole system, with its data, on your cloud or any sandbox. Agents run in it, get evaluated in it, and models train against it. We never own your compute or your data. --- # Quickstart Source: https://denv.ai/docs/tutorials/quickstart/ Section: tutorials By the end of this page you will have a running environment with a database and an agent box, a fork of it made in about a second, and a reward produced by a verifier that ran apart from the agent. Everything runs on one Linux machine with Docker; a laptop works for the walkthrough. You need: a Linux host (or a Mac for the control plane with a Linux box for the sidecar), Docker, and Go 1.22+ to build the binaries until the hosted release is live. ## 1. Get the binaries ``` git clone https://github.com/opencrew-ai/denv-backend cd denv-backend && make build export PATH="$PWD/bin:$PATH" ``` You now have `denv-cp` (the control plane), `denvd` (the sidecar that runs on each machine), and `denv` (the CLI). ## 2. Start a control plane For the managed service you would skip this step and run `denv login`. For a local walkthrough, run your own: ``` DENV_BOOTSTRAP_KEY=dak_local denv-cp -addr :8080 -db ./denv.db -public-url http://127.0.0.1:8080 & export DENV_SERVER=http://127.0.0.1:8080 DENV_API_KEY=dak_local denv whoami ``` `whoami` prints the team the key belongs to. Keys that start with `dak_` are team API keys. ## 3. Connect a machine A target is where environments run. The simplest target is the machine you are on. Mint a join token and start the sidecar with it: ``` TOKEN=$(denv host join-token | python3 -c 'import json,sys;print(json.load(sys.stdin)["token"])') denvd join --server $DENV_SERVER --token $TOKEN --state ./host.json --data ./data & denv host list ``` The host shows `"connected": true` and reports its storage: `dir` on a plain disk, `zfs` when you pass `--zfs-pool`. Forks are copies on `dir` and clones on `zfs`; both are correct, one is about a second. On a real server you would run the one-line installer instead, which does the same and installs a service: ``` curl -fsSL https://get.denv.ai | sh -s -- --token $TOKEN --server $DENV_SERVER ``` ## 4. Describe the environment Create a project and write a compose file. This one is a world: Postgres with data, Redis, and an agent box that is never root, with a verifier that grades a file the agent writes. ``` denv project create --kind rl-environment quickstart cat > compose.yaml <<'EOF' x-denv: kind: rl-environment agent: service: box verifier: image: busybox:1.36 command: ["sh", "-c", "[ \"$(cat /submission/home/agent/answer.txt)\" = 42 ] && echo 1 > /logs/verifier/reward.txt || echo 0 > /logs/verifier/reward.txt"] collect: ["/home/agent/answer.txt"] services: db: image: postgres:16-alpine environment: { POSTGRES_PASSWORD: world } volumes: ["pg:/var/lib/postgresql/data"] healthcheck: { test: ["CMD-SHELL", "pg_isready -U postgres"], interval: 2s, retries: 30 } box: image: python:3.12-slim user: "1000:1000" command: ["sleep", "infinity"] environment: { HOME: /home/agent, PGHOST: db, PGPASSWORD: world } volumes: ["home:/home/agent"] depends_on: [db] volumes: { pg: {}, home: {} } EOF ``` Three things in this file are denv's: `x-denv.agent.service` names the box the agent gets, and it must set a non-root `user`; `x-denv.verifier` is the grader, which runs after the agent is stopped on a copy of the `collect` paths; everything else is ordinary compose. Egress is denied by default. ## 5. Deploy ``` denv deploy --project quickstart -f compose.yaml --wait ``` You will see the phases run: `resolve`, `place`, `prepare`, `start`, `health`, `snapshot`, `verdict`. The snapshot phase freezes writers, checkpoints Postgres, and captures every volume at the same instant. The result is a version; the environment `alpha` now points at it. ``` ENV=$(denv project envs quickstart | python3 -c 'import json,sys;print([e for e in json.load(sys.stdin)["environments"] if e["Name"]=="alpha"][0]["ID"])') denv env get $ENV ``` ## 6. Act as the agent, then fork Write something as the agent (note the uid: the box is not root) and leave a row in the database: ``` denv exec --service box $ENV -- sh -c 'id -u; echo 42 > /home/agent/answer.txt' denv exec --service db $ENV -- psql -U postgres -c "create table t(x int); insert into t values (1)" ``` Now capture the live state and fork it: ``` denv snapshot $ENV denv fork --name branch-1 $ENV BRANCH=$(denv project envs quickstart | python3 -c 'import json,sys;print([e for e in json.load(sys.stdin)["environments"] if e["Name"]=="branch-1"][0]["ID"])') denv exec --service db $BRANCH -- psql -U postgres -tAc "select count(*) from t" ``` The fork has the row. It has its own network and its own copies of every volume; nothing you do in it reaches `alpha`. ## 7. Grade ``` denv verify $ENV ``` Watch the phases: `stop` freezes the agent box, `collect` copies `/home/agent/answer.txt` to a root-owned directory and refuses symlinks, pipes and devices, `snapshot` and `clone` make a fork with the agent still frozen, `grade` runs the busybox verifier there, `verdict` records `reward: 1`. The agent's box stays frozen so you can inspect what it did; pass `--resume` to unfreeze it. Try the attack a reinforcement-learning policy would find: ``` denv verify --resume $ENV >/dev/null denv exec --service box $ENV -- sh -c 'ln -sf /etc/passwd /home/agent/answer.txt' denv verify --resume $ENV ``` The verdict shows `reward: 0`, `source: refused`, and the reason `home/agent/answer.txt: symlink`. The grader never ran. ## 8. Clean up ``` denv env destroy $BRANCH denv env destroy $ENV ``` ## Where next - [Run a Harbor task on denv](/docs/how-to/harbor/) uses the same environment as a Harbor backend. - [Connect a target](/docs/how-to/connect-a-target/) covers Hetzner, AWS and hosted sandbox providers. - [Concepts](/docs/explanation/concepts/) explains projects, environments, deployments and versions. - [CLI reference](/docs/reference/cli/) lists every command. --- # Connect a target Source: https://denv.ai/docs/how-to/connect-a-target/ Section: how-to A target is where environments run. Connect at least one; you can connect several and pick per environment. denv never owns the compute: machines are in your account, data stays on their disks. ## Your own machines Any Linux host with Docker. Mint a join token, then run the installer as root on the host: ``` denv host join-token ``` ``` curl -fsSL https://get.denv.ai | sh -s -- --token --server https://api.denv.ai ``` The installer downloads `denvd` for the host's architecture, installs a systemd unit, and joins. The token is single-use and becomes a per-host secret. `denvd` opens one outbound WebSocket and never listens on a port, so no inbound firewall rule is needed. With a ZFS pool, snapshots are atomic across volumes and forks are clones: ``` curl -fsSL https://get.denv.ai | sh -s -- --token --server https://api.denv.ai --zfs-pool denv ``` Check what the host reported: ``` denv host list ``` `storage` is `zfs` or `dir`. On `dir`, forks copy; on `zfs`, forks take about a second regardless of data size. ## A cloud account denv provisions in Connect the account once; the `provision` phase of a deploy creates a machine when no connected host has capacity, installs `denvd` through cloud-init, and the machine joins by itself. Hetzner: ``` denv cloud connect hetzner --name hz --token --server-type ax41-nvme --location fsn1 ``` AWS: not yet available; your own EC2 machines work today through the installer above. Machines carry a `denv` label and are deletable from your own console at any time. ## A hosted sandbox provider Connect the provider with your key. No sidecar; the control plane drives the provider's API. ``` denv cloud connect e2b --name e2b --token denv cloud connect daytona --name daytona --token ``` Each connected provider appears as a target: ``` curl -s $DENV_SERVER/v1/targets -H "Authorization: Bearer $DENV_API_KEY" ``` The response carries a capability row per target. Hosted providers run one container per environment: a compose file with more than one service fails at `prepare` with a clear message. Snapshot and fork are available where the provider has a primitive; `verify` is not available on hosted providers yet. The E2B and Daytona drivers are untested against the live providers as of this writing. To run a sandbox on a specific provider: ``` curl -s -X POST $DENV_SERVER/v1/targets/e2b/sandboxes -H "Authorization: Bearer $DENV_API_KEY" \ -H 'Content-Type: application/json' -d '{"image":"python:3.12-slim","cmd":["sleep","infinity"]}' ``` ## Choosing where an environment runs Deploys prefer a connected sidecar host. A project can be pinned to a named cloud so every environment of it lands there. Sandboxes created through the API can name a target explicitly, as above. --- # Deploy from GitHub on push Source: https://denv.ai/docs/how-to/push-to-deploy/ Section: how-to ## 1. Install the denv GitHub App on the repository From the console, *Targets → GitHub → Install*, or install the app directly from GitHub and pick the repositories. The app needs Contents read, Metadata read, and Checks write. ## 2. Link the installation to your team After installing, GitHub redirects back with an installation id. Link it: ``` curl -s -X POST $DENV_SERVER/v1/github/installations//link -H "Authorization: Bearer $DENV_API_KEY" curl -s $DENV_SERVER/v1/github/installations -H "Authorization: Bearer $DENV_API_KEY" ``` ## 3. Create the project from the repository ``` denv project create --kind rl-environment --source github:you/ops-world ops-world curl -s -X POST $DENV_SERVER/v1/projects/ops-world/deploy-from-source -H "Authorization: Bearer $DENV_API_KEY" ``` `deploy-from-source` reads `docker-compose.yaml` from the default branch and deploys `alpha`. Change the branch or the file path: ``` curl -s -X POST $DENV_SERVER/v1/projects/ops-world/source -H "Authorization: Bearer $DENV_API_KEY" \ -H 'Content-Type: application/json' -d '{"default_branch":"main","compose_path":"envs/ops/compose.yaml"}' ``` ## 4. Push Every push to the default branch deploys `alpha` from that commit. A check run named `denv` appears on the commit, moves to success or failure, and links to the deployment's phase log. The commit sha is recorded with the deployment. Other branches deploy only when you ask: either an environment named after the branch already exists (`feature/x` becomes `feature-x`), or the project opts in: ``` curl -s -X POST $DENV_SERVER/v1/projects/ops-world/source -H "Authorization: Bearer $DENV_API_KEY" \ -H 'Content-Type: application/json' -d '{"deploy_branches":true}' ``` ## What happens on a failed push If the compose file is missing or refused at `resolve`, the check run fails with the reason and no deployment is created. If a later phase fails, the deployment is `rolled_back` to the previous healthy version and the check run says so. --- # Run a Harbor dataset on denv Source: https://denv.ai/docs/how-to/harbor/ Section: how-to ## 1. Install the package ``` pip install "denv[harbor]" export DENV_SERVER=https://api.denv.ai DENV_API_KEY=dak_… ``` ## 2. Run ``` harbor run --dataset terminal-bench/terminal-bench-2 --agent claude-code -e denv.harbor:DenvEnvironment ``` Nothing about the dataset or the agent changes. For a local task directory: ``` harbor run -p ./tasks/my-task -a oracle -e denv.harbor:DenvEnvironment ``` ## What the task's config maps to | In the task | On denv | |---|---| | `[environment] docker_image` | a sandbox from that image with `/app`, `/logs` and the task `workdir` persisted | | `environment/docker-compose.yaml` | the file deployed as a project; the agent runs in the first service | | `allow_internet = true` | `egress: allow`; otherwise denied and proven by a probe | | only a `Dockerfile` | not yet available: set `docker_image` to a prebuilt image | The verifier runs where Harbor expects it and writes `reward.txt` into `/logs/verifier`, which Harbor downloads into the trial directory. ## Verify it works The backend repository runs Harbor's smallest task through the adapter in its conformance suite: ``` make e2e-frameworks # in denv-backend; needs Docker and python/.venv ``` The `harbor` check runs the oracle agent on a fixture task and expects reward 1. --- # Use denv as an Inspect sandbox Source: https://denv.ai/docs/how-to/inspect/ Section: how-to ## 1. Install the package ``` pip install "denv[inspect]" export DENV_SERVER=https://api.denv.ai DENV_API_KEY=dak_… ``` The sandbox registers itself through Inspect's entry point; no import is needed in your task file. ## 2. Describe the sandbox in `denv.yaml` Inspect treats a string sandbox config as a file path, so put the configuration in a file next to the task. From an image: ```yaml image: python:3.12-slim egress: deny volumes: [/work] ``` From a version of a whole environment, data included: ```yaml from_version: ver_01… service: opsbox ``` ## 3. Select it in the task ```python from inspect_ai import Task, task from inspect_ai.dataset import json_dataset from inspect_ai.scorer import includes from inspect_ai.solver import generate, use_tools from inspect_ai.tool import bash @task def ops_triage(): return Task( dataset=json_dataset("tasks.jsonl"), solver=[use_tools(bash()), generate()], scorer=includes(), sandbox=("denv", "denv.yaml"), ) ``` ``` inspect eval ops_triage.py --model openai/gpt-5 ``` Each sample gets its own environment at `sample_init` and it is destroyed at `sample_cleanup`. Only the `volumes` paths persist across snapshot and fork; the image's writable layer does not. ## Grade with a separate verifier From a scorer, call the verifier declared in the environment's compose file: ```python from denv import Client v = Client().verify(env_id) score = v.reward # None if the verify failed ``` The agent's processes are frozen, the declared paths are collected with special files refused, and the grader runs on a fork. The environment id for the sample is available on the sandbox object as `_env_id`. --- # Run an OpenEnv environment on denv Source: https://denv.ai/docs/how-to/openenv/ Section: how-to ## 1. Install the package ``` pip install "denv[openenv]" export DENV_SERVER=https://api.denv.ai DENV_API_KEY=dak_… ``` ## 2. Use the provider ```python from denv.openenv import DenvProvider from my_env import MyEnv env = MyEnv.from_docker_image("ghcr.io/you/my-env:v1", provider=DenvProvider(egress="allow")) obs = env.reset() obs = env.step(action) env.close() ``` `egress="allow"` is required today because the environment's HTTP port must be published and an egress-denied network cannot publish ports. The ingress proxy that lifts this is not yet available. ## 3. Fork an episode ```python root = MyEnv.from_docker_image("ghcr.io/you/my-env:v1", provider=DenvProvider(egress="allow")) root.reset() for a in prefix: root.step(a) branches = [root.provider.fork() for _ in range(8)] ``` `fork()` snapshots the live environment as a new version and returns a provider bound to a new environment created from it, with its own base URL. What persists is what the environment keeps in its `volumes` (default `/work`); process memory does not survive a fork, so the OpenEnv server restarts and must rebuild in-memory state from disk. --- # Grade an episode with a separate verifier Source: https://denv.ai/docs/how-to/verify/ Section: how-to ## 1. Declare the agent service and the verifier ```yaml x-denv: kind: rl-environment agent: service: box # must set a non-root user verifier: image: ghcr.io/you/grader:1.2 # or `service: ` to grade inside a service of the fork command: ["/grade.sh"] collect: ["/home/agent/answer.txt", "/home/agent/out"] timeout_sec: 300 ``` `collect` paths are absolute paths inside the agent service. They are copied to a root-owned directory before anything is graded. ## 2. Write the grader The grader sees the copy at `/submission`, read-only, and writes the reward: ```sh #!/bin/sh # /grade.sh inside ghcr.io/you/grader:1.2 mkdir -p /logs/verifier if diff -q /submission/home/agent/answer.txt /expected.txt >/dev/null; then echo 1 > /logs/verifier/reward.txt else echo 0 > /logs/verifier/reward.txt fi ``` If `reward.txt` is absent, the last line of stdout is parsed as the reward. The grader runs on the fork's network, so it can query the fork's database too; it runs as root in its own container, with the agent's processes already stopped. ## 3. Run it ``` denv verify denv verify --resume # unfreeze the agent afterwards ``` From Python: ```python from denv import Client v = Client().verify(env_id) print(v.reward, v.source, v.refused) ``` ## What the record shows | Field | Meaning | |---|---| | `reward` | the number the grader produced; `null` if the verify failed | | `source` | `reward.txt`, `stdout`, or `refused` | | `refused` | paths that were not regular files or directories, with the reason: `symlink`, `fifo`, `device`, `socket`, or over the size and depth limits. Any refusal scores 0 and the grader does not run | | `missing` | declared paths that did not exist | | `agent_frozen` | whether the agent's container was left frozen for inspection | The grader's fork is destroyed at the verdict. The agent's environment is left as it was. --- # Snapshot and fork Source: https://denv.ai/docs/how-to/snapshot-and-fork/ Section: how-to ## Fork from the current version ``` denv fork --name rollout-017 ``` A fork is a new environment created from a version. Its name is yours; its parent is the version. On ZFS hosts it is a clone and takes about a second regardless of data size. ## Snapshot mid-episode, then branch ``` denv snapshot denv fork --name rollout-017-b ``` `snapshot` captures the live state as a new version: databases are checkpointed, writers are frozen for the capture, every volume is captured at the same instant, writers resume and health is re-checked. The environment then points at the new version, so the next fork starts from it. From Python, `Client.fork(env_id)` snapshots first by default; pass `live=False` to fork the last version without a new snapshot. ## Fork for one episode through the sandbox API ``` curl -s -X POST $DENV_SERVER/v1/sandboxes -H "Authorization: Bearer $DENV_API_KEY" \ -H 'Content-Type: application/json' -d '{"from_version":"ver_01…","name":"ep-42"}' ``` Returns when healthy. Destroy it when the episode ends: ``` denv env destroy ``` ## Roll back ``` denv env versions denv rollback ver_01… ``` Rollback restores the version's snapshot and re-checks health. The spec is the version's, so it is a full return to that state. ## What persists Named volumes in the compose file are what a snapshot captures and a fork carries. For sandboxes created from an image, the paths listed in `volumes` (default `/work`) are volumes; anything written elsewhere lives in the container's writable layer and is not carried by a fork. --- # denv CLI Source: https://denv.ai/docs/reference/cli/ Section: reference The `denv` binary is a thin client over the REST API: every verb maps to one endpoint and prints the JSON response, indented. Errors print `denv: ` to stderr and exit 1; usage errors exit 2. ## Invocation rules - **Flags go before positional arguments.** `denv fork --name f1 alpha` works; `denv fork alpha --name f1` treats `--name` as a positional and fails. This is Go's `flag` package behaviour and applies to every command. - Subcommands (`project create`, `env get`, `files put`, …) take their own flags after the subcommand name. - Commands that start a deployment (`deploy`, `fork`, `snapshot`, `verify`, `rollback`) default to `--wait`, which polls the deployment and prints one line per phase as it changes, then `deployment : `. A `failed` deployment exits 1. Pass `--wait=false` to print the `202` response and return. - The HTTP client timeout is 5 minutes per request. ## Configuration | Source | Precedence | Notes | |---|---|---| | `DENV_API_KEY` | first | A team API key (`dak_…`) or a user session token (`dst_…`). | | `DENV_SERVER` | first | Base URL of the control plane. Default `http://localhost:8080`. | | `~/.config/denv/config.json` | fallback | Written by `denv login`. Read only when `DENV_API_KEY` is unset. Its `server` is used only when `DENV_SERVER` is also unset. | | `XDG_CONFIG_HOME` | — | When set, the config file is `$XDG_CONFIG_HOME/denv/config.json`. | The config file has three keys: `server`, `api_key`, `team` (the team id the key was approved for). It is written with mode `0600`. With no credential, any command prints `denv: not signed in: run `denv login` or set DENV_API_KEY`. ## Commands ### login ``` denv login [--server URL] [--no-browser] ``` Runs the device flow: asks the server for a code, prints the console URL and the user code, opens the browser at the `verification_uri_complete` URL unless `--no-browser` is given, then polls `POST /v1/device/token` every `interval` seconds until a key is issued or the code expires (15 minutes). On success it writes the config file and prints `Signed in to team . Key saved to `. `--server` defaults to `DENV_SERVER` (or the saved server, or `http://localhost:8080`). This command parses its own two flags and ignores anything else. ### whoami ``` denv whoami ``` `GET /v1/me`. Prints `{team, teams[], user?}`. ### templates ``` denv templates [--kind K] ``` `GET /v1/templates?kind=K`. Prints the known kinds (`rl-environment`, `agent`) and the templates, each with its compose text. ### init ``` denv init [--out compose.yaml] ``` Fetches the templates and writes the compose file of the first template whose name or kind matches the argument to `--out` (default `compose.yaml`). Refuses to overwrite an existing file (` exists; pass --out to write elsewhere`). When the template has a warm command, prints the `denv deploy … --warm` invocation to use. Templates today: `rl-environment-ops` (kind `rl-environment`) and `agent-runtime` (kind `agent`). ### project ``` denv project create [--kind K] [--source github:org/repo] denv project list denv project envs ``` | Subcommand | Endpoint | Flags | |---|---|---| | `create` | `POST /v1/projects` | `--kind` (`rl-environment`, the default, or `agent`); `--source` free text, conventionally `github:owner/repo` | | `list` | `GET /v1/projects` | — | | `envs` | `GET /v1/projects/{project}/envs` | `` is a name or an id | Creating a project also creates its `alpha` environment. ### cloud ``` denv cloud connect [--name N] [--token T] [--server-type T] [--location L] denv cloud list ``` `connect` posts `{provider, name, config}` to `POST /v1/clouds`, where `config` holds only the flags that were set, under the keys `token`, `server_type`, `location`. Providers compiled in: `manual`, `hetzner`, `e2b`, `daytona`. The provider validates the token before the cloud is recorded. For `e2b` and `daytona` the cloud becomes a driver host (a target) the next time targets are listed; see [Targets](targets.md). Those two providers are untested against the live provider. `list` is `GET /v1/clouds`; secrets in `config` are returned as `•••`. ### host ``` denv host list denv host join-token ``` `list` is `GET /v1/hosts` (each host with a `connected` flag). `join-token` is `POST /v1/join-tokens` with an empty body; the response includes `token` (`djt_…`, single use, valid 1 hour), `expires_in_sec`, and `command`, a ready-to-paste install line. ### deploy ``` denv deploy --project P [--env alpha] [-f compose.yaml] [--warm "cmd args"] [--wait] ``` Reads the compose file and posts `{env, compose, warm}` to `POST /v1/projects/{project}/deploy`. `--project` is required. `--warm` is split on whitespace into the warm command's argv. Prints the deployment id, then the phases. ### env ``` denv env get denv env versions denv env deployments denv env destroy ``` `get`, `versions`, `deployments` are the corresponding `GET /v1/envs/{env}…` routes. `destroy` is `DELETE /v1/envs/{env}`; it prints the destroy deployment and does not wait. `` is always an environment id (`env_…`), never a name. ### fork ``` denv fork [--name N] [--version V] ``` `POST /v1/envs/{env}/fork` with `{name, version}`. `--version` defaults to the environment's current version; `--name` defaults to `-fork-`. Waits by default. ### snapshot ``` denv snapshot ``` `POST /v1/envs/{env}/snapshot`. Captures live state as a new version of the same environment. Waits by default. ### verify ``` denv verify [--resume] ``` `POST /v1/envs/{env}/verify` with `{resume}`. Freezes the agent service, collects the declared paths, snapshots, forks, grades on the fork. Without `--resume` the agent service stays paused afterwards. The reward is in the `verdict` phase's evidence. See [Deployments](deployments.md) and [Compose](compose.md). Waits by default. ### files ``` denv files put --service S denv files get --service S ``` `put` tars the local file or directory (named by its base name) and uploads it with `PUT /v1/envs/{env}/files?service=S&path=`; the archive is extracted into that directory. The archive limit is 32 MiB. `get` downloads `` as a tar via `GET /v1/envs/{env}/files` and extracts regular files and directories under ``, printing ` files written under `. `--service` is required for both. ### rollback ``` denv rollback ``` `POST /v1/envs/{env}/rollback` with `{version}`. The version must be a healthy snapshot of that environment. Waits by default. ### exec ``` denv exec --service S -- ``` `POST /v1/envs/{env}/exec` with `{service, cmd}`. The `--` separator is optional; everything after `` is the argv. The server-side default timeout is 120 s. Prints `{exit_code, stdout, stderr}`. A non-zero exit code is not a CLI error. ### deployment ``` denv deployment get denv deployment watch ``` `get` prints `{deployment, phases[]}`. `watch` polls every 700 ms and prints phases as they change, as the `--wait` commands do. ### events ``` denv events ``` `GET /v1/events?subject=` where the subject is `env:`, `dep:`, or `host:`. Returns up to 200 events, newest first. ## Not yet available - `denv promote`, `denv prove`, `denv sheet`, `denv export`, `denv import`: described in the blog, not in the binary. - Project names are accepted everywhere an id is for projects; environments and deployments require ids. --- # denvd sidecar Source: https://denv.ai/docs/reference/denvd/ Section: reference `denvd` is the one process denv runs on a customer's machine. It is a static binary with four verbs. It opens one outbound WebSocket to the control plane and never listens on a port. It runs environments with Docker Compose, keeps their datasets on disk (or in a ZFS pool), and executes commands idempotently. ## Verbs ``` denvd join --server URL --token T [--state FILE] [--data DIR] [--zfs-pool POOL] [--env-file FILE] denvd run [--server URL] [--state FILE] [--data DIR] [--zfs-pool POOL] denvd install --server URL --token T [--data DIR] [--zfs-pool POOL] [--env-file FILE] denvd version ``` | Verb | What it does | |---|---| | `join` | Runs in the foreground. On first connect it presents the join token; the control plane assigns a host id and a host secret, which are saved to the state file. Later connects use the saved identity. | | `run` | Same loop as `join` but refuses to start without a saved identity; a token is not required. | | `install` | Linux, root only. Copies itself to `/usr/local/bin/denvd`, writes `/etc/denv/denvd.env` (mode 0600) with `DENV_SERVER`, `DENV_JOIN_TOKEN`, `DENV_DATA_DIR`, `DENV_ZFS_POOL`, writes `/etc/systemd/system/denvd.service`, then runs `systemctl daemon-reload` and `systemctl enable --now denvd`. | | `version` | Prints the build version (set at build time; `dev` otherwise). | ## Flags and environment Every flag falls back to an environment variable. | Flag | Env | Default | Meaning | |---|---|---|---| | `--server` | `DENV_SERVER` | — (required) | Control plane base URL, `http://` or `https://`. The stream URL is derived: `wss:///v1/hosts/stream`. | | `--token` | `DENV_JOIN_TOKEN` | — | Join token (`djt_…`). Required on the first join unless `run` is used. Never persisted. | | `--state` | — | `/etc/denv/host.json` as root, else `~/.denv/host.json` | Identity file: `server`, `host_id`, `secret`, `data_dir`, `zfs_pool`. Mode 0600. | | `--data` | `DENV_DATA_DIR` | `/var/lib/denv` as root, else `~/.denv/data` | Data directory. | | `--zfs-pool` | `DENV_ZFS_POOL` | — | ZFS pool name. Storage becomes `zfs` only when the pool exists (`zpool list ` succeeds). | | `--env-file` | — | — | Reads `KEY=VALUE` lines into the environment before the flags above are resolved (used by cloud-init). | Saved state wins for `server`, `data_dir`, and `zfs_pool` unless the flag is given. ## What the sidecar does ### Connect and identify On each session the sidecar sends a `hello` frame with its host id, its secret (or the join token when it has no id), and the machine's capabilities: hostname, OS, arch, CPU count, memory, free disk under the data dir, storage backend (`zfs` or `dir`), Docker server version, whether `docker compose` exists, whether network namespaces are usable (Linux as root), and the denvd version. A heartbeat frame is sent every 10 s. Reconnects back off from 1 s to 30 s. A `PolicyViolation` close from the control plane (bad token, unknown host, rejected secret) is fatal and the process exits. ### Execute commands Each command carries a command id. Results are written to `/commands/.json` before being sent; a command id seen again is answered from that file without re-executing. One command runs at a time per environment. Command kinds and what they do on the host are listed in [Protocol](protocol.md). In short: `env.create` renders a compose file and starts the stack, `env.snapshot` freezes containers and captures every volume, `env.fork` clones a snapshot into a new environment, `env.rollback` restores a snapshot, `env.exec` runs `docker compose exec`, `env.put`/`env.get` move tar archives with `docker cp`, `verify.collect`/`verify.run` implement grading, `env.destroy` stops the stack and removes its datasets. ### Data directory layout | Path | Contents | |---|---| | `/envs//compose.yaml` | The rendered compose file for the environment. | | `/envs//spec.json` | Services, volumes, egress, saved so rollback can re-render after a restart. | | `/envs//vol//` | Volume data (`dir` storage). | | `/snaps///` | Snapshot copies (`dir` storage). | | `/commands/.json` | The stored result of every command (idempotency log). | | `/verify//submission/` | Root-owned copy of the collected paths for one verify. | | `/verify//logs/` | The grader's `/logs/verifier` mount. | With ZFS the volumes are datasets `/envs//vol/` mounted at `//envs//vol/`, snapshots are ZFS snapshots named after the version, and forks are clones. ### Storage backends | Backend | Reported as | Snapshot | Fork | Rollback | |---|---|---|---|---| | ZFS pool | `zfs` | `zfs snapshot` per volume dataset | `zfs clone` | `zfs rollback` | | Directory | `dir` | Recursive copy of `vol/` into `snaps/` | Recursive copy out of `snaps/` | Recursive copy over `vol/` | `dir` is correct and slow; it is the backend on a laptop and on any Linux host without a pool. ### Compose project naming The Docker Compose project name is the environment id in lower case. The stack's network is `_default`. ### What the sidecar refuses The compose subset is enforced by the control plane before a spec reaches the sidecar (see [Compose](compose.md)). On the host, the sidecar additionally refuses, during verify, any collected path that is a symlink, FIFO, device, or socket, and trees that exceed 256 MiB, 20,000 files, or 32 directory levels. ## Install script ``` curl -fsSL https://get.denv.ai | sh -s -- --token [--server URL] [--zfs-pool POOL] [--data DIR] ``` The script (`scripts/install.sh` in the backend repository) requires Linux, root, and Docker on the host. It detects `amd64` or `arm64`, downloads `/denvd-linux-` with `curl` or `wget`, installs it to `/usr/local/bin/denvd` (`DENV_BIN_DIR` overrides), and runs `denvd install --server … [--token …] [--zfs-pool …] [--data …]`. | Flag | Env | Default | |---|---|---| | `--server` | `DENV_SERVER` | `https://api.denv.ai` | | `--releases` | `DENV_RELEASES` | `https://get.denv.ai` | | `--token` | `DENV_JOIN_TOKEN` | — | | `--zfs-pool` | `DENV_ZFS_POOL` | — | | `--data` | `DENV_DATA_DIR` | — | The control plane also serves the binaries itself at `GET /denvd` (amd64) and `GET /denvd-linux-{amd64|arm64}` from its releases directory; the `command` field returned by `POST /v1/join-tokens` uses that route. ## systemd unit Written by `denvd install`: ``` [Unit] Description=denv sidecar After=network-online.target docker.service Wants=network-online.target [Service] EnvironmentFile=/etc/denv/denvd.env ExecStart=/usr/local/bin/denvd join --state /etc/denv/host.json Restart=always RestartSec=3 LimitNOFILE=1048576 [Install] WantedBy=multi-user.target ``` The join token in `/etc/denv/denvd.env` is consumed on the first connect; after that the identity in `/etc/denv/host.json` is what authenticates the host, and the token line is ignored (a used token is rejected by the control plane, but `join` only presents it when no identity is saved). ## Provisioned machines When the control plane provisions a machine (Hetzner today), the cloud-init it passes installs `ca-certificates curl docker.io docker-compose-v2 zfsutils-linux`, writes `/etc/denv/join.env` with `DENV_SERVER` and `DENV_JOIN_TOKEN`, downloads `denvd` from the control plane's denvd URL, and runs `denvd install --env-file /etc/denv/join.env`. It does not create a ZFS pool; such a host reports `storage: dir` unless a pool is configured afterwards. --- # REST API Source: https://denv.ai/docs/reference/api/ Section: reference Base URL: the control plane (`https://api.denv.ai` for the managed service, `http://localhost:8080` locally). All request and response bodies are JSON unless noted. Response field names for stored objects are the Go struct field names (`ID`, `Name`, `CurrentVersionID`, …); request field names are lower snake case. ## Authentication | Credential | Header | Used by | |---|---|---| | Team API key `dak_…` | `Authorization: Bearer dak_…` | CLI, CI, Python adapters. Minted by the device flow or `POST /v1/keys`. Revoked keys are refused. | | User session `dst_…` | `Authorization: Bearer dst_…` plus optional `X-Denv-Team: ` | Console. Issued by GitHub sign-in, valid 30 days. Without the header, the user's first team is used. | | None | — | `/health`, `/auth/*`, device code and token, `/denvd*`. | | Host secret / join token | inside the WebSocket `hello` frame | `/v1/hosts/stream` (sidecars only). | | Webhook signature | `X-Hub-Signature-256` | `/webhooks/github`. | Auth columns below: **key** (API key or session), **user** (session only), **none**. ### Errors Every error is `{"error": ""}` with an HTTP status. Common statuses: `400` bad input, `401` bad credential, `403` session without a team or not a member, `404` not found or not in your team, `409` conflict (duplicate name, used token), `413` body too large, `428` device authorization pending, `502` the host or an upstream refused the command, `503` feature not configured on this server, `504` sandbox did not become healthy in time. ### CORS When `DENV_CONSOLE_URL` is set, `/v1/*` and `/auth/*` answer CORS for that origin and for `http://localhost:*` / `http://127.0.0.1:*`, allowing `Authorization, Content-Type, X-Denv-Team` and `GET, POST, PUT, DELETE, OPTIONS`. `OPTIONS` returns `204`. The examples below assume: ```sh export DENV_SERVER=http://localhost:8080 DENV_API_KEY=dak_… ``` ## Auth, teams, and keys ### GET /auth/github/start — none Query `return` (optional): where to send the browser after sign-in. Must be the console origin or a loopback URL; default `/app/`. Redirects (`302`) to GitHub with scopes `read:user user:email`. `503` when GitHub sign-in is not configured. ```sh curl -i "$DENV_SERVER/auth/github/start?return=http://localhost:3100/app/" ``` ### GET /auth/github/callback — none GitHub's redirect target. Creates or updates the user, gives a first-time user a personal team named after their login, mints a session, and redirects to `#token=dst_…`. `400` on unknown or expired state. ### POST /auth/dev-login — none Only when the server runs with `DENV_DEV_LOGIN=1`; otherwise `404`. Body `{email, name?}`. Response `{token: "dst_…"}`. ```sh curl -s -X POST $DENV_SERVER/auth/dev-login -H 'Content-Type: application/json' -d '{"email":"you@example.com"}' ``` ### POST /v1/device/code — none Starts a CLI login. Response: ```json {"device_code":"ddc_…","user_code":"XXXX-XXXX","verification_uri":"/app/device/","verification_uri_complete":"/app/device/?code=XXXX-XXXX","expires_in":900,"interval":5} ``` ### POST /v1/device/approve — user Body `{user_code, team_id, name?}`. The caller must be a member of `team_id`. Mints an API key named `name` (default `cli`) and marks the code approved. Response `{approved: true, key_id, team_id}`. ### POST /v1/device/token — none Body `{device_code}`. `428 {"error":"authorization_pending"}` until approved; `400 {"error":"invalid_device_code"}` for an unknown code; `410` if the approved key was already handed out. Response `{api_key: "dak_…", team: {ID, Name, CreatedAt}}`. The key is returned exactly once. ### GET /v1/me — key Response `{team, teams: [{ID, Name, CreatedAt, Role}], user?}`. With an API key, `teams` has one entry with role `key`; with a session, `user` is present and `teams` lists memberships. ```sh curl -s $DENV_SERVER/v1/me -H "Authorization: Bearer $DENV_API_KEY" ``` ### GET /v1/teams — user Response `{teams: [{ID, Name, CreatedAt, Role}]}`. ### POST /v1/teams — user Body `{name}`. Creates a team with the caller as owner. Response `201 {team}`. ### GET /v1/keys — key Response `{keys: [{ID, TeamID, Name, CreatedBy, CreatedAt, LastUsedAt, RevokedAt}]}`. Key material is never returned. ### POST /v1/keys — key Body `{name?}` (default `key`). Response `201 {key: {…}, api_key: "dak_…"}`; `api_key` is shown once. ```sh curl -s -X POST $DENV_SERVER/v1/keys -H "Authorization: Bearer $DENV_API_KEY" -H 'Content-Type: application/json' -d '{"name":"ci"}' ``` ### DELETE /v1/keys/{id} — key Revokes a key of the caller's team. Response `{revoked: true}`; `404` if not found. ## Projects ### GET /v1/templates — key Query `kind` (optional). Response `{kinds: ["rl-environment","agent"], templates: [{name, kind, description, requires?, compose, warm?}]}`. ### GET /v1/projects — key Response `{projects: [{ID, TeamID, Name, Kind, Source, CreatedAt}]}`. ### POST /v1/projects — key Body `{name, kind?, source?}`. `kind` is `rl-environment` (default) or `agent`. Creates the project and its `alpha` environment. Response `201 {project, environments: [alpha]}`; `409` on a duplicate name. ```sh curl -s -X POST $DENV_SERVER/v1/projects -H "Authorization: Bearer $DENV_API_KEY" -H 'Content-Type: application/json' \ -d '{"name":"ops-world","kind":"rl-environment","source":"github:you/ops-world"}' ``` ### GET /v1/projects/{project}/envs — key `{project}` is an id or a name. Response `{environments: [{ID, ProjectID, Name, Stage, HostID, ParentVersionID, CurrentVersionID, Status, CreatedAt, Endpoints}]}`. ### POST /v1/projects/{project}/deploy — key Body `{env?, compose, warm?}`. `env` defaults to `alpha`; `compose` is the compose file text; `warm` is an argv array run after health. The compose file is parsed and validated here (`400` with the reason on refusal). Response `202 {deployment, phases: [...]}`. ```sh curl -s -X POST $DENV_SERVER/v1/projects/ops-world/deploy -H "Authorization: Bearer $DENV_API_KEY" -H 'Content-Type: application/json' \ -d "$(jq -n --arg c "$(cat compose.yaml)" '{env:"alpha",compose:$c}')" ``` ### GET /v1/projects/{project}/source — key Response `{source: {ProjectID, Repo, DefaultBranch, ComposePath, InstallationID, DeployBranches, LastDeployedSHA}}`. ### POST /v1/projects/{project}/source — key Body `{default_branch?, compose_path?, installation_id?, deploy_branches?}`. Empty strings and `0` leave a field unchanged; `deploy_branches` is a nullable boolean. `installation_id` must be an installation linked to the team. Response `{source}`. ### POST /v1/projects/{project}/deploy-from-source — key Deploys from GitHub. Body `{branch?, env?}`: `branch` defaults to the repo's default branch; `env` defaults to `alpha` for the default branch, otherwise the branch name with every character outside `[a-z0-9]` replaced by `-`. Requires the GitHub App (`503` otherwise) and a linked installation that can see the repo (`409` otherwise). Response `202 {deployment, phases, branch, env}`. ## Environments ### GET /v1/envs/{env} — key Response `{environment, connected}` where `connected` reports whether the environment's host currently has a stream (or is a driver host). ### GET /v1/envs/{env}/versions — key Response `{versions: [{ID, ProjectID, EnvID, Number, Spec, Snapshot, Healthy, CreatedAt}]}`. ### GET /v1/envs/{env}/deployments — key Response `{deployments: [{ID, ProjectID, EnvID, Kind, Status, VersionID, TargetEnvID, Params, Error, CreatedAt, FinishedAt}]}`. ### POST /v1/envs/{env}/fork — key Body `{name?, version?}`. `version` defaults to the current version and must be a healthy snapshot. Response `202 {deployment, phases}`; the new environment's id is `deployment.EnvID`. ```sh curl -s -X POST $DENV_SERVER/v1/envs/env_01…/fork -H "Authorization: Bearer $DENV_API_KEY" -H 'Content-Type: application/json' -d '{"name":"rollout-01"}' ``` ### POST /v1/envs/{env}/snapshot — key No body. Captures live state as a new version of the same environment. Response `202 {deployment, phases}`; the version id is `deployment.VersionID`. ### POST /v1/envs/{env}/rollback — key Body `{version}`. The version must be a healthy snapshot of this environment. Response `202 {deployment, phases}`. ### POST /v1/envs/{env}/verify — key Body `{resume?}`. Requires the environment's version to declare `x-denv.agent.service` and `x-denv.verifier` (`400` otherwise). Response `202 {deployment, phases}`. Read the reward from `GET /v1/deployments/{dep}`: the `verdict` phase's `Evidence` has `reward`, `source`, `exit_code`, `refused`, `missing`, `agent_frozen`, `took_ms`. ### POST /v1/envs/{env}/exec — key Body `{service, cmd: [...], timeout_sec?}` (default 120 s). Response `200 {exit_code, stdout, stderr}`. A non-zero exit code is still `200`; `502` means the host could not run it. ```sh curl -s -X POST $DENV_SERVER/v1/envs/env_01…/exec -H "Authorization: Bearer $DENV_API_KEY" -H 'Content-Type: application/json' \ -d '{"service":"opsbox","cmd":["sh","-c","id -u"]}' ``` ### PUT /v1/envs/{env}/files — key Query `service`, `path` (both required). Body: a tar archive (`Content-Type: application/x-tar`), at most 32 MiB (`413` otherwise). Extracted into `path` inside the service. Response `{ok: true, bytes}`. ```sh tar -C ./fixtures -cf - . | curl -s -X PUT "$DENV_SERVER/v1/envs/env_01…/files?service=opsbox&path=/home/ubuntu/fixtures" \ -H "Authorization: Bearer $DENV_API_KEY" -H 'Content-Type: application/x-tar' --data-binary @- ``` ### GET /v1/envs/{env}/files — key Query `service`, `path`. Response: a tar archive of `path` (`application/x-tar`). A missing path is `502` with the host's error. ### DELETE /v1/envs/{env} — key Starts a destroy deployment. Response `202 {deployment}`. ## Deployments and sandboxes ### GET /v1/deployments/{dep} — key Response `{deployment, phases: [{DeploymentID, Seq, Name, Status, Evidence, Error, StartedAt, FinishedAt}]}`. `Evidence` is a JSON object whose keys depend on the phase; see [Deployments](deployments.md). ### POST /v1/sandboxes — key One environment for one episode, returned when healthy. Body, one of two shapes: | Field | Type | Meaning | |---|---|---| | `from_version` | string | Fork this healthy version (whole stack, data included). | | `image` | string | Or: one container from this image. | | `cmd` | string[] | Command for the container (image form). | | `env` | object | Environment variables (image form). | | `ports` | string[] | Container ports to publish (image form). Needs `egress: allow`. | | `volumes` | string[] | Container paths that persist and fork; default `["/work"]` (image form). | | `egress` | `deny` \| `allow` | Default `deny`. | | `project` | string | Project for image sandboxes; default `sandboxes`, created on demand. | | `name` | string | Environment name; default `sbx-`. | | `timeout_sec` | int | How long to wait for health; default 600. | Response `201 {environment, deployment}`; `502` if the deployment failed; `504` on timeout. ```sh curl -s -X POST $DENV_SERVER/v1/sandboxes -H "Authorization: Bearer $DENV_API_KEY" -H 'Content-Type: application/json' \ -d '{"image":"python:3.12-slim","cmd":["sleep","infinity"]}' ``` ## Targets, clouds, and hosts ### GET /v1/clouds — key Response `{clouds: [{ID, TeamID, Provider, Name, Config, CreatedAt}]}`; `token`, and any config key containing `secret` or `key`, is replaced by `•••`. ### POST /v1/clouds — key Body `{provider, name?, config?}`. Providers: `manual`, `hetzner` (config `token`, `server_type` default `cpx31`, `location` default `nbg1`, `image` default `ubuntu-24.04`), `e2b` and `daytona` (config `token`; untested against the live provider). The provider validates the config first (`400`). Response `201 {cloud}`. ### POST /v1/join-tokens — key Body `{cloud_id?}`. Response `201 {token: "djt_…", expires_in_sec: 3600, command}`. ### GET /v1/hosts — key Response `{hosts: [{ID, TeamID, CloudID, Name, Status, ProviderRef, LastSeenAt, CreatedAt, Capabilities, connected}]}`. `Capabilities` is the sidecar's report (see [Protocol](protocol.md)) or the provider's capability row. ### GET /v1/targets — key Hosts of both kinds. Response `{targets: [{…host fields, kind, provider?, connected, capabilities, table?}]}` where `kind` is `sidecar` or `driver:` and `table` is the static capability row for a hosted provider. Sidecars are listed first. ### POST /v1/targets/connect — key Body `{provider, token?, name?, config?}` for a hosted sandbox provider (`e2b`, `daytona`). Validates the token, records the cloud, and creates its driver host in one step. Response `201 {cloud, host, info, capabilities}`. ### POST /v1/targets/{name}/sandboxes — key `POST /v1/sandboxes` pinned to the cloud named `{name}`. Image form only (`400` otherwise); `404` if no such cloud. Response `201 {environment, deployment, target}`. ### GET /v1/hosts/stream — sidecar WebSocket endpoint for `denvd`. Not for API clients. ## GitHub ### POST /webhooks/github — webhook Verified with `X-Hub-Signature-256` against `DENV_GITHUB_WEBHOOK_SECRET` (`401` on mismatch; `503` when the app is not configured; body limit 2 MiB). Handles `installation` (created, unsuspend, new_permissions_accepted, deleted), `push`, `ping`. A push to a branch deploys every project whose source is that repo: the default branch to `alpha`; other branches to the branch-named environment only if it already exists or the project has `deploy_branches` on. Response `{ok, sha, branch, projects: [{project, deployment?, env?, skipped?, error?}]}`. ### GET /v1/github/installations — key Response `{installations: [{InstallationID, TeamID, AccountLogin, CreatedAt}], configured}`. ### POST /v1/github/installations/{id}/link — key Binds an installation the app has reported to the caller's team. `404` if the installation has not been seen; `409` if already linked elsewhere. Response `{installation}`. ### GET /v1/github/repos?installation={id} — key Repositories the linked installation can see. Response `{repos: [{full_name, default_branch, private}]}`. ## Releases ### GET /denvd, GET /denvd-linux-amd64, GET /denvd-linux-arm64 — none The sidecar binary from the server's releases directory (`DENV_RELEASES_DIR`, default `bin`). `/denvd` is amd64 (`?arch=arm64` selects the other). `404` when that build is not present on the server. ## Events ### GET /v1/events?subject=… — key `subject` is `env:`, `dep:`, `host:`, or `cmd:` (progress lines). Up to 200 events, newest first, filtered to the caller's team. Response `{events: [{ID, TeamID, Subject, Kind, Body, At}]}`. ### GET /health — none `{"status":"ok"}`. --- # Compose file and x-denv Source: https://denv.ai/docs/reference/compose/ Section: reference An environment is described by a Docker Compose file plus an `x-denv` block. The control plane parses it at deploy time, rejects anything outside the subset below with the reason shown, and pins the result as the version's spec. The sidecar renders a new compose file from that spec; your file is never run verbatim. ## Supported keys Top level: `services`, `volumes`, `x-denv`. Everything else is ignored. Per service: | Key | Handling | |---|---| | `image` | Required. | | `environment` | Map or `KEY=VALUE` list. | | `ports` | Container ports only. `"8080:80"` is reduced to `80`; a `/tcp` suffix is dropped. Host ports are never pinned because forks need unique ports. | | `volumes` | `":"` only, where `` is declared at top level. | | `depends_on` | List or map (keys only; conditions ignored). | | `command` | String (split on whitespace) or list. | | `healthcheck` | `test` (string or list), `interval`, `retries`. | | `user` | `uid[:gid]`. | | `mem_limit`, `pids_limit`, `cpus` | Caps; see below. | | `deploy.resources.limits.{memory,cpus,pids}` | Alternative spelling of the caps; used only where the shorthand is absent. | ## Refused | Condition | Error | |---|---| | No services | `compose: no services` | | Service without `image` | `compose: service "x" has no image (build is not supported in v0; push an image)` | | `network_mode: host` | `compose: service "x" uses network_mode: host, which cannot be forked` | | `privileged` or `cap_add` | `compose: service "x" asks for privileged or cap_add, which is refused` | | Bind mount (`/path:…` or `./path:…`) | `compose: service "x" bind-mounts "/path"; only named volumes are forkable` | | Undeclared volume | `compose: service "x" uses undeclared volume "v"` | | Volume without a container path | `compose: service "x" volume "v" must be :` | | `x-denv.egress` not `deny`/`allow` | `compose: x-denv.egress must be deny or allow, got "…"` | `build:` is not supported: a service needs a pushed image. ## Resource caps Every service gets memory, pid, and CPU caps. Zero in the file means the sidecar default. | Cap | Compose keys | Default | |---|---|---| | Memory | `mem_limit` (`2g`, `512m`, `1024k`, bytes) or `deploy.resources.limits.memory` | 4096 MiB | | Processes | `pids_limit` or `deploy.resources.limits.pids` | 2048 | | CPUs | `cpus` or `deploy.resources.limits.cpus` | 2.0 | The sidecar renders these as `mem_limit`, `pids_limit`, `cpus` on every service of the generated compose file. ## Network With `x-denv.egress: deny` (the default) the stack's default network is rendered `internal: true`: no route out, and published ports are not reachable from the host. A probe container on that network tries `http://1.1.1.1/` before the environment is reported healthy; the result is `denied (verified)`. If the probe reaches the internet the deploy fails with `egress is denied but the stack reached the internet`. With `allow`, ports are published on `127.0.0.1:` and reported as the environment's `Endpoints`. ## x-denv ```yaml x-denv: kind: rl-environment # informational; the project's kind is authoritative egress: deny # deny (default) | allow warm: # argv run in the first service after health, before the snapshot - psql - -f - /seed/orders.sql agent: service: opsbox # the service the agent runs in; must not be root verifier: image: ghcr.io/you/grader:1 # run as its own container on the fork's network … # service: opsbox # … or inside this service of the fork, as root (exactly one of image/service) command: ["/grade.sh"] # required collect: ["/home/ubuntu/answer.txt"] # absolute paths in the agent service timeout_sec: 600 # default 600 ``` | Key | Rule | |---|---| | `kind` | Parsed, not enforced. | | `egress` | `deny` or `allow`. | | `warm` | List of strings; also settable per deploy by the API's `warm` field, which overrides the file. Runs in the first service (sorted by name), timeout 600 s; a non-zero exit fails the deploy. | | `agent.service` | Must name a service. That service must set a non-root `user` (an empty `user` counts as root). Error: `compose: x-denv.agent.service "x" must set a non-root user (user: "1000:1000"); the agent is never root`. | | `verifier` | Exactly one of `image` or `service` (`compose: x-denv.verifier needs exactly one of image or service`). `command` required (`compose: x-denv.verifier.command is required`). `collect` paths must be absolute (`compose: x-denv.verifier.collect path "x" must be absolute`). Requires `agent.service` (`compose: x-denv.verifier requires x-denv.agent.service, the service whose paths are collected`). | Services are processed in sorted name order; the "first service" (for `warm` and for the post-snapshot health check) is the alphabetically first. ## Verifier contract `denv verify` runs the verifier on a fork of the environment, after the agent service is frozen and the `collect` paths have been copied to a root-owned directory on the host. **Image verifier.** `docker run --rm --user 0` on the fork's network, with: | Mount / env | Value | |---|---| | `/submission` | the collected copy, read-only; the collected paths keep their absolute layout under it (`/submission/home/ubuntu/answer.txt`) | | `/logs/verifier` | writable; write the reward here | | `SUBMISSION` | `/submission` | | `DENV_VERIFY_ID` | the verify deployment id | **Service verifier.** The copy is placed at `/submission` inside the named service of the fork, `/logs/verifier` is created there, and the command runs with `docker exec --user 0` and the same two environment variables. `reward.txt` is copied back out afterwards. **Reward.** Read, in order: `/logs/verifier/reward.txt` parsed as a float; else the last whitespace-separated token of stdout parsed as a float. Neither present: the grade phase fails with `verifier wrote no reward (exit N)`. The verdict records `source` as `reward.txt`, `stdout`, `refused`, or `error`. **Refusal.** The copy is checked before any grader runs. Anything below sets the reward to `0` with `source: refused`, drops the whole copy, and skips the grader: | Finding | Recorded as | |---|---| | symlink | `: symlink` | | FIFO | `: fifo` | | device | `: device` | | socket | `: socket` | | deeper than 32 levels | `: deeper than 32` | | more than 256 MiB in total | `: submission larger than 256 MiB` | | more than 20,000 files | `: more than 20000 files` | A declared path that does not exist is recorded under `missing` and does not cause a refusal. The grader's run is limited to `timeout_sec` (default 10 minutes); exceeding it fails the grade phase. ## Example ```yaml x-denv: kind: rl-environment egress: deny agent: { service: opsbox } verifier: image: busybox:1.36 command: ["sh", "-c", "mkdir -p /logs/verifier; [ \"$(cat /submission/home/agent/answer.txt)\" = hello ] && echo 1 > /logs/verifier/reward.txt || echo 0 > /logs/verifier/reward.txt"] collect: ["/home/agent/answer.txt"] services: opsbox: image: python:3.12-slim user: "1000:1000" command: ["sleep", "infinity"] environment: { HOME: /home/agent } volumes: ["home:/home/agent"] mem_limit: 2g pids_limit: 512 db: image: redis:7-alpine volumes: ["rd:/data"] healthcheck: { test: ["CMD", "redis-cli", "ping"], interval: 2s, retries: 15 } volumes: { home: {}, rd: {} } ``` ## Not yet available `x-denv.clock`, `x-denv.verifier` for hosted sandbox providers (E2B, Daytona refuse `verify`), and host-side image builds are described in the blog but not implemented. A compose file with no declared volumes still deploys; its snapshot is an empty tree. --- # Python package Source: https://denv.ai/docs/reference/python/ Section: reference ```sh pip install "denv[all]" # or denv[harbor], denv[inspect], denv[openenv] export DENV_SERVER=https://api.denv.ai DENV_API_KEY=dak_… ``` Package `denv`, version 0.1.0, Python ≥ 3.11. Dependencies: `httpx`, `pydantic`, `pyyaml`. Optional extras pull `harbor`, `inspect-ai`, `openenv-core`. The package registers the entry point `inspect_ai = denv._inspect_registry` so Inspect finds the sandbox without an import. ## denv.Client ```python from denv import Client c = Client(server=None, api_key=None, timeout=900.0) ``` | Argument | Default | Meaning | |---|---|---| | `server` | `DENV_SERVER`, else `http://localhost:8080` | Control plane base URL. | | `api_key` | `DENV_API_KEY` | Required; `RuntimeError("DENV_API_KEY is not set")` otherwise. | | `timeout` | 900 s | Default HTTP timeout. | All methods are synchronous. Any HTTP error raises `RuntimeError("denv : ")`. ### Sandboxes and versions | Method | Returns | Behaviour | |---|---|---| | `sandbox(*, image=None, from_version=None, cmd=None, env=None, ports=None, egress="deny", volumes=None, name=None, project=None, timeout_sec=600)` | `Sandbox` | `POST /v1/sandboxes`. One of `image` or `from_version` is required (`ValueError`). Returns when healthy. | | `destroy(env_id)` | `None` | `DELETE /v1/envs/{id}`. | | `current_version(env_id)` | `str` | The environment's `CurrentVersionID`. | | `snapshot(env_id, timeout_sec=300)` | `str` | `POST …/snapshot`, waits for the deployment, returns the new version id. | | `fork(env_id, name=None, timeout_sec=600, live=True)` | `Sandbox` | With `live=True` (default) snapshots first, then creates a sandbox from that version; with `live=False` forks the current version. | | `verify(env_id, resume=False, timeout_sec=900)` | `Verdict` | `POST …/verify`, waits, reads the verdict phase's evidence. Raises `RuntimeError` if the deployment failed. | ### Inside an environment | Method | Returns | Behaviour | |---|---|---| | `exec(env_id, service, cmd: list[str], timeout_sec=120)` | `ExecResult` | `POST …/exec`. | | `put_tar(env_id, service, path, tar_bytes)` | `None` | `PUT …/files`; the archive is extracted into `path`. | | `get_tar(env_id, service, path)` | `bytes` | `GET …/files`; a tar of `path`. | | `put_file(env_id, service, source, target_path)` | `None` | Uploads one local file to the file path `target_path`. | | `put_bytes(env_id, service, target_path, data)` | `None` | Writes `data` to the file path `target_path`. | | `put_dir(env_id, service, source_dir, target_dir)` | `None` | Uploads a directory's contents into `target_dir`. | | `get_file(env_id, service, source_path, target)` | `None` | Downloads one file to the local path `target`. | | `get_bytes(env_id, service, source_path)` | `bytes` | The first regular file of the archive; `FileNotFoundError` if none. | | `get_dir(env_id, service, source_dir, target_dir)` | `None` | Extracts the directory's contents under `target_dir` (`docker cp src/.` semantics). | ### Dataclasses ```python @dataclass class Sandbox: id: str; name: str; status: str endpoints: dict[str, str] # service → "127.0.0.1:port" (only with egress allow) raw: dict # the environment object as returned @dataclass class ExecResult: exit_code: int; stdout: str; stderr: str @dataclass class Verdict: deployment_id: str reward: float | None # None when the deployment failed refused: list[str] # ": " missing: list[str] source: str # reward.txt | stdout | refused | error status: str # succeeded | failed | rolled_back evidence: dict # the verdict phase's evidence as returned ``` ### shell_command ```python from denv.client import shell_command argv = shell_command(script, *, cwd=None, env=None, user=None) -> list[str] ``` Builds one argv that runs `script` through `bash -c` when the image has bash and `sh -c` otherwise, with `env` exported first and `cwd` applied; a non-root `user` wraps it in `su -s /bin/sh -c`. Used by the adapters. ### Example ```python from denv import Client c = Client() sbx = c.sandbox(image="python:3.12-slim", cmd=["sleep", "infinity"]) c.put_bytes(sbx.id, "main", "/work/a.txt", b"alpha") print(c.exec(sbx.id, "main", ["cat", "/work/a.txt"]).stdout) # alpha child = c.fork(sbx.id) # carries /work c.destroy(child.id); c.destroy(sbx.id) ``` ## Harbor adapter ``` harbor run --dataset --agent -e denv.harbor:DenvEnvironment ``` `denv.harbor.DenvEnvironment` subclasses Harbor's `BaseEnvironment`. `type()` returns `"denv"`; `capabilities` reports `disable_internet=True, docker_compose=True` (not `mounted`, so Harbor downloads logs). | Task input | What the adapter does | |---|---| | `environment/docker-compose.yaml` (or `.yml`) present | The file is deployed as environment `harbor-` of a project named `sandboxes` (created if missing). The agent runs in service `main`. | | `[environment] docker_image` set | `sandbox(image=…, cmd=["sleep","infinity"], volumes=["/app","/logs", workdir?])`. | | Only a `Dockerfile` | Refused at `_validate_definition`: `denv: this task has only a Dockerfile. Set [environment] docker_image in task.toml (or add a compose file) until denv ships host-side builds.` | | `allow_internet = true` | `egress: allow`; otherwise `deny`. | After start, `/logs/agent /logs/verifier /logs/artifacts /tests /solution /app` are created in the service. `exec(command, cwd, env, timeout_sec, user)` runs through `shell_command` with Harbor's merged env and resolved user (default timeout 600 s). `upload_file/upload_dir/download_file/download_dir` map to the client's `put_file/put_dir/get_file/get_dir`. `stop(delete=True)` destroys the environment; `delete=False` leaves it. Not mapped: `cpus`, `memory_mb`, `storage_mb`, `gpus` from `task.toml` (the sidecar defaults apply); fork reuse across `-k` attempts; a separate-verifier mode. Compose tasks always use service `main` as the agent service. ## Inspect adapter ```python Task(..., sandbox="denv") # image from DENV_SANDBOX_IMAGE, default python:3.12-slim Task(..., sandbox=("denv", "denv.yaml")) # a config file Task(..., sandbox=("denv", DenvSandboxConfig(image="…"))) # inline Task(..., sandbox=("denv", "ver_01…")) # fork of a version ``` Inspect treats a string config as a file path, so an image name must go in a file or a `DenvSandboxConfig`; a bare non-file string that does not start with `ver_` raises `ValueError`. A `denv.yaml` or `denv.yml` next to the task is picked up automatically (`config_files()`). `denv.yaml` keys (`DenvSandboxConfig`): | Key | Default | Meaning | |---|---|---| | `image` | — | One container from this image, command `sleep infinity`. | | `from_version` | — | Or: fork this version. | | `egress` | `deny` | `deny` or `allow`. | | `service` | `main` | Service `exec` runs in. | | `volumes` | `["/work"]` | Paths that persist across snapshot and fork (image form). | Behaviour: `task_init` constructs a `Client` so a missing key fails before any model call; `sample_init` creates one sandbox per sample and returns `{"default": DenvSandbox}`; `sample_cleanup` destroys it; `exec(cmd, input, cwd, env, user, timeout)` joins the argv with `shlex.quote`, writes `input` to `/tmp/.denv_stdin` and redirects it, runs through `shell_command` (default timeout 300 s) and returns Inspect's `ExecResult`; `write_file`/`read_file` use `put_bytes`/`get_bytes`; `connection()` returns the command `denv exec --service -- sh`; `default_concurrency()` is 8. `task_cleanup` and `cli_cleanup` are no-ops. ## OpenEnv adapter ```python from denv.openenv import DenvProvider env = MyEnv.from_docker_image("org/my-env:v1", provider=DenvProvider()) ``` `DenvProvider(client=None, egress="deny")` implements OpenEnv's `ContainerProvider`: | Method | Behaviour | |---|---| | `start_container(image, port=None, env_vars=None, **kwargs)` | `sandbox(image, cmd=kwargs.get("cmd"), env, ports=[port or 8000], egress)`. Sets and returns `base_url = "http://"`. Raises `RuntimeError` when no endpoint was published, which is the case with `egress="deny"` because a denied network is internal; pass `egress="allow"`. | | `wait_for_ready(base_url, timeout_s=30)` | Polls `GET /health` until the status is below 500. | | `fork()` | `client.fork(env_id)` (snapshot then fork) and returns a new `DenvProvider` bound to the child, `base_url` set if the child published an endpoint. | | `stop_container()` / `close()` | Destroy the environment. | Not yet available: `from_denv_version`, `denv export --openenv`. ## Conformance suite ```sh make e2e-frameworks # all checks ./scripts/e2e-frameworks.sh verify harbor # a subset ``` `scripts/e2e-frameworks.sh` starts a control plane and a sidecar on the local machine (Docker and `python/.venv` required) and runs `python/tests/conformance.py`. Checks: `client` (exec, files, fork carries `/work`, fork is isolated), `verify` (correct answer scores 1, a symlink scores 0 and is refused, a FIFO scores 0, the agent stays frozen without `resume`), `inspect` (an eval with `mockllm/model`, accuracy 1.0, plugin found via the entry point), `harbor` (the `oracle` agent on a fixture task, reward 1), `openenv` (`python -m http.server` as the image, ready, stopped). The script exits non-zero with the sidecar and control-plane logs on failure. --- # Control plane ↔ sidecar protocol Source: https://denv.ai/docs/reference/protocol/ Section: reference Each `denvd` keeps one outbound WebSocket to `GET /v1/hosts/stream` on the control plane. Every message in either direction is one JSON frame. The control plane sends commands; the sidecar answers with acks, progress lines, results, and unsolicited events. Nothing on the control plane ever dials a host. ## Frame ```json { "type": "command", // hello | command | ack | progress | result | event | heartbeat "host_id": "host_…", "command_id": "cmd_…", // commands, acks, progress, results "kind": "env.create", // commands; events use it as the event kind "env_id": "env_…", // commands addressed to an environment "payload": { … }, // kind-specific "ok": true, // results only "error": "…", // results, when ok is false "message": "…", // progress lines "sent_at": "2026-10-12T00:00:00Z" } ``` ## Session 1. The sidecar dials `wss:///v1/hosts/stream` (`ws://` for an `http://` server). The read limit is 16 MiB per frame. 2. Its first frame is `hello` with payload `{host_id, token, capabilities}`. On the first ever connect `host_id` is empty and `token` is the join token (`djt_…`); afterwards `host_id` is set and `token` is the host secret (`dhs_…`). 3. The control plane admits the host. A new host consumes the join token (single use, `409`-style rejection when used or expired), is assigned an id, and receives one `hello` frame back with `{host_id, secret}`, which it persists. A known host has its secret hash checked. Any refusal closes the socket with WebSocket status 1008 (policy violation); the sidecar treats that as fatal and exits. 4. The sidecar sends a `heartbeat` frame every 10 seconds; the control plane records `last_seen_at` on each. The grace window on the control plane is 60 seconds. 5. When the stream drops, the host is marked `unreachable` and the sidecar reconnects with backoff from 1 s to 30 s. Reconnecting as the same host replaces any older connection. ### Capabilities Sent in `hello` and returned by `host.report`: | Field | Meaning | |---|---| | `hostname`, `os`, `arch`, `cpus`, `memory_mb`, `disk_gb` | The machine. `disk_gb` is free space under the data dir. | | `storage` | `zfs` when a pool is configured and present, else `dir`. | | `docker` | Docker server version, `""` if absent. | | `compose` | Whether `docker compose` exists. | | `netns` | Linux and root. | | `public_ip` | Optional. | | `denvd_version` | Build version. | ## Command lifecycle A command is recorded in the control plane's `commands` table before it is written to the stream, so a crash between the two leaves a `sent` row rather than a silent loss. The sidecar replies `ack` when it has the frame, zero or more `progress` frames while working, then exactly one `result` with `ok` and either `payload` or `error`. The control plane records progress lines as events on subject `cmd:` and stores the result on the command row. **Idempotency.** The sidecar writes every result to `/commands/.json` before sending it. A command id it has already completed is answered from that file without executing again, so re-delivery after a reconnect is safe. One command runs at a time per environment on a host. **Timeouts.** The control plane bounds each phase at 15 minutes (orchestrator `CommandTimeout`); `env.exec` uses the request's `timeout_sec` plus 10 s. A timed-out command is marked failed on the control plane; the sidecar may still finish it and store the result. ## Command kinds ### host.report Payload `{}`. Result `{capabilities, environments, load_avg}` (the sidecar today fills `capabilities` only). ### env.create Payload: ```json {"version":"ver_…","services":[Service…],"volumes":["pg","home"],"from_snapshot":"","warm":["cmd","arg"],"egress":"deny"} ``` Prepares one dataset per volume, renders a compose file, runs `docker compose up -d --wait --force-recreate --quiet-pull --remove-orphans`, then runs the egress probe. Result: `{endpoints: {service: "127.0.0.1:port"}, started_in, digests: {service: image@sha256}, storage, egress}` where `egress` is `allowed` or `denied (verified)`. The `warm` field is carried in the payload but the warm command itself is sent by the control plane as a separate `env.exec` in the `warm` phase. **Service** (as rendered by the control plane): | Field | Meaning | |---|---| | `name`, `image` | Image is the reference from the compose file; digests are read back after `up`. | | `env`, `ports`, `volumes` (`":"`), `health` (`{test, interval, retries}`), `depends_on`, `command`, `user` | As in the compose file. | | `resources` | `{memory_mb, pids, cpus}`; zero means the sidecar default (4096 MiB, 2048, 2.0). | ### env.snapshot Payload `{name, keep_paused?: ["service"]}`. Checkpoints databases (Postgres `CHECKPOINT`, Redis `BGSAVE`, MySQL `FLUSH TABLES`), pauses every container that is not already paused, captures every volume with the storage backend under `name`, then unpauses everything except `keep_paused` and waits for health (90 s). Result `{snapshot, datasets, frozen_for, consistent: true}`. ### env.fork Payload `{from_env, from_snapshot, services, volumes, egress}`. Addressed to the **new** environment id. Clones the snapshot's volumes into the new environment, renders, starts, probes egress. Result `{forked_in, endpoints}`. ### env.rollback Payload `{snapshot}`. Stops the stack, restores every volume from the snapshot, re-renders from the saved spec, starts. Result `{snapshot, endpoints}`. ### env.exec Payload `{service, cmd: [...], timeout_sec?}` (default 120 s). Runs `docker compose exec -T `. Result `{exit_code, stdout, stderr}`; a non-zero exit is still `ok: true`. ### env.put Payload `{service, path, tar_b64}`. Creates `path` in the service and extracts the archive there with `docker cp`. Result `{path}`. ### env.get Payload `{service, path}`. Result `{tar_b64}`: a tar of `path`, limited to 32 MiB. ### verify.collect Payload `{id, agent_service, collect: ["/abs/path"]}`. Pauses the agent service, copies each path with `docker cp -a=false` (symlinks are not followed) into `/verify//submission/`, then walks the copy with `lstat`. Result `{files, bytes, missing: [...], refused: [": "]}`. When anything is refused the whole copy is removed. The agent service stays paused. ### verify.run Addressed to the grader's fork. Payload `{id, verifier: {image|service, command, collect, timeout_sec}, refused}`. With `refused: true` returns `{reward: 0, source: "refused"}` without running anything. Otherwise runs the verifier (see [Compose](compose.md), verifier contract). Result `{reward, exit_code, stdout, stderr, source, ran_for}`; `stdout`/`stderr` are tails. ### env.destroy Payload `{}`. `docker compose down` and removal of the environment's datasets. Result `{destroyed: ""}`. ### Declared, not implemented `egress.set`, `clock.set`, `ingress.set` exist as kinds in the protocol package; neither the control plane sends them nor the sidecar handles them (the sidecar answers `unknown command kind`). ## Events Frames of type `event` from the sidecar are stored on subject `host:` with `kind` as the event kind. The sidecar does not emit any today. ## Driver hosts A host of kind `driver:` has no stream. The hub runs its commands in-process through the provider's driver, records them in the same `commands` table, and keeps the driver's per-environment state (provider sandbox id, snapshot id) in `host_env_state`. The payloads and results are the same shapes as above; a command the provider has no primitive for fails with `: : not supported on yet` or a more specific reason. See [Targets](targets.md). --- # Deployments and phases Source: https://denv.ai/docs/reference/deployments/ Section: reference A deployment is one change to an environment, run as an ordered list of phases. Each phase ends `ok`, `failed`, or `skipped` and records a JSON evidence object. The last phase is always `verdict`. `GET /v1/deployments/{id}` returns the deployment and its phases; `denv deployment watch` prints them as they change. ## Statuses | Deployment | Meaning | |---|---| | `pending` | Created, not started. | | `running` | Phases executing. | | `succeeded` | Every non-skipped phase ok. | | `failed` | A phase failed and nothing was reverted. | | `rolled_back` | A deploy failed and the environment was restored to its previous version. | | Phase | Meaning | |---|---| | `pending`, `running`, `ok`, `failed`, `skipped` | `skipped` is used both for phases a kind does not need this time (e.g. `provision` when a host exists) and for every phase after a failure. | ## Failure rule When a phase fails, the remaining phases up to `verdict` are marked `skipped`, the deployment's `Error` is set to `: `, and `verdict` runs anyway so the outcome is recorded. For a **deploy** with a previous healthy version and a host, `verdict` sends `env.rollback` to that version and the status becomes `rolled_back`; otherwise the environment's status becomes `failed`. For every other kind the verdict records the failure and changes nothing else. Each phase is bounded by the orchestrator's command timeout (15 minutes). ## deploy `POST /v1/projects/{project}/deploy`, `deploy-from-source`, `POST /v1/sandboxes` (image form), push webhooks. | # | Phase | Evidence | Notes | |---|---|---|---| | 1 | `resolve` | `{version, number, services, volumes}` | Creates the version row from the parsed spec; remembers the previous current version for rollback. | | 2 | `place` | `{host, reason}` | `environment already on host`, `target `, `first connected host in team`, or `no connected host; provisioning`. Sidecars are preferred over provider hosts. | | 3 | `provision` | `{cloud, provider, ref, ip, name}` | Skipped (`host already placed`) when a host was found. Otherwise creates a machine through the first non-`manual` cloud; fails with `no connected host and no cloud that can provision: …` when there is none. | | 4 | `join` | `{host, joined_in_ms}` | Skipped when placed. Waits up to 6 minutes for any host of the team to connect. | | 5 | `prepare` | `{version, endpoints}` | Sends `env.create`; the sidecar prepares datasets, starts, waits for health, probes egress. | | 6 | `start` | `{started_in_ms}` | Derived from the `env.create` result. | | 7 | `health` | `{endpoints}` | Derived from the `env.create` result. | | 8 | `warm` | `{exit}` or `{exit, stderr}` | Skipped (`no warm command`) unless `warm` was given. Runs it in the first service with a 600 s timeout; non-zero exit fails. | | 9 | `snapshot` | `{snapshot, datasets, frozen_for, consistent}` | `env.snapshot` named after the version id. | | 10 | `ingress` | `{endpoints, names: "pending ingress proxy"}` | Always skipped today. | | 11 | `verdict` | `{version, snapshot, took_ms}` on success; `{reverted_to}`, `{reverted: "no previous version"}`, or `{reverted: "rollback failed: …"}` on failure | Marks the version healthy and makes it the environment's current version. | ## fork `POST /v1/envs/{env}/fork`, `POST /v1/sandboxes` (`from_version`). Creates a new environment (stage `fork`, parent version set) on the source's host. | # | Phase | Evidence | |---|---|---| | 1 | `place` | `{host, reason: "same host as source snapshot"}`; fails if the source's host is not connected. | | 2 | `clone` | `{forked_in_ms, from}` from `env.fork`. | | 3 | `health` | `{endpoints}`, derived. | | 4 | `verdict` | `{env, took_ms}`; the fork's current version is the source version. | ## snapshot `POST /v1/envs/{env}/snapshot`. Creates a new version with the same spec as the current one. | # | Phase | Evidence | |---|---|---| | 1 | `snapshot` | `{snapshot, datasets, frozen_for, consistent}`. | | 2 | `health` | `{answered: true}`: `true` executed in the first service after unpause. | | 3 | `verdict` | `{version, snapshot, took_ms}`; on failure `{kept: }` and nothing changes. | ## rollback `POST /v1/envs/{env}/rollback`. The version must be a healthy snapshot of this environment. | # | Phase | Evidence | |---|---|---| | 1 | `restore` | `{snapshot}` from `env.rollback`; fails if the host is not connected. | | 2 | `health` | `{restored: }`, derived. | | 3 | `verdict` | `{version}`; the environment's current version is set. | ## verify `POST /v1/envs/{env}/verify`. Requires `x-denv.agent.service` and `x-denv.verifier` in the current version's spec. Creates a new version row for the capture. | # | Phase | Evidence | Notes | |---|---|---|---| | 1 | `stop` | `{agent_service, mode: "pause"}` | Recorded only; the freeze happens inside `collect` so there is no gap. | | 2 | `collect` | `{files, bytes, missing, refused}` | `verify.collect`: agent paused, paths copied to a root-owned directory, special files refused. | | 3 | `snapshot` | `{snapshot, datasets, frozen_for, consistent}` | Agent kept paused unless `resume`; the version is marked healthy. | | 4 | `clone` | `{grader_env, forked_in_ms}` | A new environment `-verify-` (stage `verify`) forked from the capture. | | 5 | `grade` | `{reward, exit_code, stdout, stderr, source, ran_for}` | `verify.run` on the grader's fork; `refused: true` when collect refused anything. | | 6 | `verdict` | `{reward, source, exit_code, refused, missing, agent_frozen, took_ms}` | The grader's fork is destroyed. On failure `reward` is `null`. | The agent's environment keeps its previous current version; the capture version exists in `versions` and is healthy, so it can be forked. The agent service remains paused unless `resume: true`. ## destroy `DELETE /v1/envs/{env}`. | # | Phase | Evidence | |---|---|---| | 1 | `teardown` | `null`; skipped with `{reason: "host not connected; datasets left in place"}` when the host is unreachable. | | 2 | `verdict` | `null`; the environment's status becomes `destroyed`. | ## promote Declared with phases `gate.fork`, `gate.consistency`, `gate.replay`, `gate.invariants`, `gate.soak`, `apply`, `verdict`. **Not yet available**: no API route starts a promotion. ## Events Every deployment appends `deployment.started` (`{kind}`) and `deployment.finished` (`{status, took_ms, error}`) to subjects `env:` and `dep:`; `provision` appends `host.provisioned`; GitHub-triggered deploys append `deployment.source` (`{sha, branch, repo}`) to `dep:`. Read them with `GET /v1/events?subject=…`. --- # Targets Source: https://denv.ai/docs/reference/targets/ Section: reference A target is a host that can take environment commands. There are two kinds. | Kind | What it is | Connection | |---|---|---| | `sidecar` | A machine running `denvd`: your own, or one denv provisioned in your cloud account. | The sidecar's outbound WebSocket. `connected` is true while the stream is open. | | `driver:` | A hosted sandbox provider (`e2b`, `daytona`) driven by the control plane with your token. | None; the provider's API. `connected` is always true. | `GET /v1/targets` lists both, sidecars first, each with `kind`, `provider` (driver hosts), `connected`, `capabilities` (the sidecar's report, or the provider's static row), and, for providers, `table`. ## Clouds and hosts A **cloud** is a credential and configuration record (`POST /v1/clouds`, `POST /v1/targets/connect`). A **host** is a target. The mapping: | Provider | Cloud config | What it produces | |---|---|---| | `manual` | none | Nothing. Hosts join with a token you run on a machine. | | `hetzner` | `token` (required), `server_type` (`cpx31`), `location` (`nbg1`), `image` (`ubuntu-24.04`) | Machines created on demand by the `provision` phase; each joins as a sidecar host with the cloud's id. | | `e2b` | `token` | One driver host named `e2b` with status `connected`. Untested against the live provider. | | `daytona` | `token` | One driver host named `daytona`. Untested against the live provider. | A driver host is created by `POST /v1/targets/connect`, or on demand for a cloud created through `POST /v1/clouds` the next time targets are listed or placement runs (`EnsureDriverHosts`). Each cloud has at most one driver host. ## Join tokens `POST /v1/join-tokens {cloud_id?}` returns a `djt_…` token valid for one hour, single use. A sidecar presents it in its first `hello`; the control plane assigns the host id and a `dhs_…` secret, records the host with the token's cloud id, and marks the token used. A used or expired token is refused with `join: join token already used` / `join: join token expired`. The response's `command` field is a paste-able install line using the control plane's own denvd download route. ## Placement For a **deploy**, the `place` phase chooses in this order: 1. The host the environment already lives on, if connected (its datasets are there). 2. The named target, when the request pins one (`POST /v1/targets/{name}/sandboxes`, or `DeployTo`): the cloud's driver host, else any connected sidecar joined through that cloud; otherwise `target "" (): no connected host`. 3. The first connected host of the team. Connected hosts are listed sidecars first, then driver hosts, so a machine you run is preferred over a provider. 4. Nothing: `provision` creates a machine through the first non-`manual` cloud, and `join` waits up to 6 minutes for it. With no such cloud the deploy fails with `no connected host and no cloud that can provision: connect a cloud or run `denvd join` on a machine`. Forks, snapshots, rollbacks, verifies, and destroys always run on the environment's own host. ## Capability table The static row returned as `table` for each hosted provider (`internal/drivers/caps.go`): | Provider | Multi-service | Snapshot | Fork | Fork time | Egress deny | Caps | GPU | Verify | Notes | |---|---|---|---|---|---|---|---|---|---| | `e2b` | no | pause (sandbox persisted) | — | n/a | provider network config | provider tier | no | no | one sandbox per environment; pause/resume keeps state, no branch | | `daytona` | no | snapshot to image | new sandbox from snapshot | tens of seconds | limited | provider tier | partial | no | — | | `modal` | no | filesystem + memory snapshot | restore from snapshot | seconds | block_network | per sandbox | yes | no | driver not shipped: Modal's sandbox API is Python-only | | `prime` | no | snapshot | new sandbox from snapshot | seconds | yes | per sandbox | yes | no | driver not shipped | A sidecar host's capabilities are what it detected (see [Protocol](protocol.md)); on a ZFS pool a fork is a clone and takes about a second, on `dir` storage it is a copy. ## What each driver does today Both drivers are implemented against the providers' public REST APIs and tested only with recorded request shapes; neither has been run against the live provider. | Command | E2B | Daytona | |---|---|---| | `env.create` | `POST /sandboxes` with the image as `templateID`; a `command` is started in the background. Single service only. | `POST /sandbox` with `image` (falls back to `snapshot`), polled until `started`; a `command` is started in the background. Single service only. | | `env.exec` | Process RPC on the sandbox's envd. | `POST /toolbox/{id}/toolbox/process/execute`. | | `env.put` / `env.get` | One file at a time through the envd files API. | One file at a time through the toolbox files API. | | `env.snapshot` | `pause` then `resume`; reports one dataset. | Refused: `Daytona's snapshot API builds images, not snapshots of a running sandbox; …` | | `env.fork` | Refused: `E2B has no fork: pause/resume continues the same sandbox and cannot branch it; use your machines or Daytona for forks` | Refused: `fork needs a sandbox snapshot, which is not enabled yet for Daytona` | | `env.destroy` | `DELETE /sandboxes/{id}` | `DELETE /sandbox/{id}?force=true` | | `verify.collect`, `verify.run` | Refused: `e2b: : not supported on e2b yet` | Refused likewise. | A multi-service spec sent to a provider fails at `prepare` with ` runs one container per environment; use your machines for multi-service stacks (N services declared)`. Provider sandboxes report no `endpoints`. ## Not yet available AWS provisioning (the control plane itself deploys on AWS; customer machines on AWS are joined manually today), Modal and Prime drivers, GPU hosts, and `verify` on any hosted provider. --- # Control plane configuration Source: https://denv.ai/docs/reference/config/ Section: reference `denv-cp` is the control plane: API server, host hub, and orchestrator in one process. It is configured by flags, each of which falls back to an environment variable. ## Flags | Flag | Env | Default | Meaning | |---|---|---|---| | `-addr` | `DENV_ADDR` | `:8080` | Listen address. | | `-db` | `DENV_DB` | `denv.db` | Database: a SQLite file path, or a `postgres://` DSN. See below for the assembled form. | | `-public-url` | `DENV_PUBLIC_URL` | `http://localhost:8080` | The URL sidecars and cloud-init use to reach this server; also the OAuth callback base and the join-token command. | | `-denvd-url` | `DENV_DENVD_URL` | `/denvd` | Where cloud-init downloads the sidecar. | | `-v` | — | off | Debug logging (request lines are logged at debug). | ## Environment variables | Variable | Meaning | |---|---| | `DENV_DB` | As `-db`. When unset and `DENV_DB_HOST` is set, a Postgres DSN is assembled from `DENV_DB_HOST`, `DENV_DB_PORT` (5432), `DENV_DB_NAME` (`denv`), `DENV_DB_USER` (`denv`), `DENV_DB_PASSWORD`, `DENV_DB_SSLMODE` (`require`). The password is redacted in logs. | | `DENV_TEAM` | Name of the first team created on an empty database (default `default`). | | `DENV_BOOTSTRAP_KEY` | API key to install for that first team. When unset a random `dak_…` is generated and printed to stderr once. Only consulted when the database has no teams. | | `DENV_CONSOLE_URL` | The console's origin. Enables CORS for it, sets the default post-login return (`/app/`), the device-flow verification URL (`/app/device/`), and the GitHub check-run link (`/app/deployment/?id=…`). | | `DENV_GITHUB_CLIENT_ID`, `DENV_GITHUB_CLIENT_SECRET` | GitHub OAuth app for console sign-in. Both unset: `/auth/github/*` answers `503`. Callback URL to register: `/auth/github/callback`. | | `DENV_DEV_LOGIN` | `1` enables `POST /auth/dev-login` (sign in by email, no GitHub). Local use only. | | `DENV_GITHUB_APP_ID`, `DENV_GITHUB_APP_PRIVATE_KEY` | GitHub App for push-to-deploy. The key is PEM text or a file path. Both unset: the app is off, `POST /webhooks/github` answers `503`, `/v1/github/*` report `configured: false`. | | `DENV_GITHUB_WEBHOOK_SECRET` | Verifies `X-Hub-Signature-256`. Unset with the app configured: webhooks are refused (logged as a warning at start). | | `DENV_GITHUB_API_URL` | Override of `https://api.github.com` (GitHub Enterprise, tests). | | `DENV_RELEASES_DIR` | Directory served by `GET /denvd*` (default `bin`; the container image sets `/releases`). | ## Database SQLite is the default and is used by the end-to-end scripts; the file is opened in WAL mode with a 5 s busy timeout. A `postgres://` DSN selects Postgres through pgx. The schema is created and migrated on every start. ## GitHub App Permissions and events the app needs: | Setting | Value | |---|---| | Repository permissions | Contents: read, Metadata: read, Checks: write (optional; a `403` is tolerated and no check runs are posted) | | Webhook URL | `/webhooks/github` | | Webhook events | `push`, `installation` | | Webhook secret | `DENV_GITHUB_WEBHOOK_SECRET` | After a user installs the app, the installation appears in `GET /v1/github/installations` and must be linked to a team with `POST /v1/github/installations/{id}/link` before pushes deploy. A project's source is `github:owner/repo`; its compose path defaults to `docker-compose.yaml`. ## CORS With `DENV_CONSOLE_URL` set, requests to `/v1/*` and `/auth/*` whose `Origin` is that URL, `http://localhost:*`, or `http://127.0.0.1:*` receive `Access-Control-Allow-Origin: `, `Access-Control-Allow-Headers: Authorization, Content-Type, X-Denv-Team`, `Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS`, `Access-Control-Max-Age: 600`. `OPTIONS` requests on those paths return `204`. Without `DENV_CONSOLE_URL` no CORS headers are sent. ## Process notes - OAuth `state` values and keys approved by the device flow are held in process memory for up to 10 and 15 minutes respectively. Run one instance of `denv-cp`, or sign-ins and `denv login` can fail when a second instance receives the follow-up request. - The bootstrap team and key are created only when the database has no teams; later teams come from GitHub sign-in or `POST /v1/teams`. - Shutdown on `SIGINT`/`SIGTERM` waits up to 10 s for in-flight requests. ## Managed service The CDK stacks under `infra/` run `denv-cp` on ECS Fargate behind an HTTPS load balancer with a 3600 s idle timeout for sidecar streams, and inject `DENV_PUBLIC_URL=https://api.denv.ai`, `DENV_CONSOLE_URL=https://denv.ai`, the database settings, and the `DENV_GITHUB_*` values from Secrets Manager (`denv/github`). See `infra/README.md` in the backend repository. --- # Protections Source: https://denv.ai/docs/reference/protections/ Section: reference These are the five checks from Karotte's public framework comparison, as denv implements them. Each is a property of the environment runtime, not of the harness running inside, so Harbor, Inspect, OpenEnv, or a custom harness gets the same behaviour. They apply to environments on sidecar hosts; hosted sandbox providers refuse `verify` and enforce caps and egress their own way (see [Targets](targets.md)). ## 1. Reading the answer key **Rule.** The agent is never root. With `x-denv.agent.service` set, the compose parser refuses a service whose `user` is empty, `0`, or `root`. **Where.** `compose.Parse`, at the `resolve` phase of a deploy (the API returns `400` before any deployment starts). **You will see.** `compose: x-denv.agent.service "opsbox" must set a non-root user (user: "1000:1000"); the agent is never root` **Limits.** Without `x-denv.agent.service` nothing enforces the user; the `rl-environment-ops` template sets `user: "1000:1000"`. Grading data is kept out of the agent's reach only insofar as the verifier runs in its own image or on a separate fork; a `service` verifier runs inside the fork's copy of a service. ## 2. Tampering with grading **Rule.** The grader never shares a process tree with the agent. `verify` pauses the agent service (`docker compose pause `) before anything is copied, copies the declared paths to a root-owned directory on the host (`/verify//submission`, mode 0700), snapshots the environment with the agent still frozen, forks it, and runs the verifier on the fork as uid 0, over the root-owned copy mounted read-only at `/submission`. **Where.** `verify.collect` and `verify.run` on the sidecar; phases `stop → collect → snapshot → clone → grade` of the verify deployment. **You will see.** Progress lines `freezing agent service `, `checking the copy`, `grading in ` or `grading inside service `; verdict evidence `{"reward": 1, "source": "reward.txt", "agent_frozen": true, …}`. After verify without `resume`, `denv exec` into the agent service fails with Docker's `is paused` error. **Limits.** The verifier image you supply is yours; denv does not inspect it. `PATH`/`LD_PRELOAD` scrubbing inside the agent container is a harness concern (Karotte does it; denv does not). ## 3. Breaking grading with a weird answer file **Rule.** The collected copy is walked with `lstat` before any grader runs. Symlinks, FIFOs, devices, and sockets are refused; so are trees over 256 MiB, over 20,000 files, or deeper than 32 levels. On any refusal the whole copy is deleted, the grader is not run, and the reward is 0. `docker cp` is invoked with `-a=false`, so a symlink is copied as a symlink and caught, not followed. **Where.** `checkTree` in the sidecar's `verify.collect`; `verify.run` with `refused: true` short-circuits. **You will see.** In the `collect` phase evidence and the verdict: `"refused": ["home/agent/answer.txt: symlink"]`, `"… : fifo"`, `"… : device"`, `"… : socket"`, `"… : deeper than 32"`, `"… : submission larger than 256 MiB"`, `"… : more than 20000 files"`; and `"source": "refused"`, `"reward": 0`. ## 4. Crashing the run by using up resources **Rule.** Every service gets `mem_limit`, `pids_limit`, and `cpus` in the rendered compose file: the values from the compose file, or 4096 MiB, 2048 pids, 2.0 CPUs when unset. A fork bomb hits the pid cap inside its own cgroup; memory pressure kills the service, not the host or the sidecar. **Where.** `stack.render` on the sidecar, from the `resources` carried in the spec. **You will see.** The caps in `/envs//compose.yaml`; the Docker error for the container when it is hit. **Limits.** No disk quota on `dir` storage; ZFS datasets have no quota set either. The host itself is not isolated from the sidecar beyond cgroups (no gVisor, no microVM). ## 5. Looking up the answer online **Rule.** `x-denv.egress` defaults to `deny`. The stack's default network is rendered `internal: true`. After the stack is healthy, and again after every fork, a `busybox` container on that network attempts `wget http://1.1.1.1/` with a 4 s timeout. The environment is reported healthy only if the probe prints `BLOCKED`. **Where.** `stack.verifyEgress` in `env.create` and `env.fork` on the sidecar. **You will see.** Progress line `healthy; egress denied (verified)` (or `healthy; egress allowed`); the `env.create` result's `"egress": "denied (verified)"`; `denv deploy` output `alpha · egress denied (verified)` in the console. If the probe reaches out, the deploy fails with `egress is denied but the stack reached the internet` and the stack is stopped. If the probe cannot run at all: `egress probe failed to run: …`. **Limits.** A denied network cannot publish ports, so `Endpoints` is empty; set `egress: allow` to publish, and then nothing is denied. DNS is not separately blocked; with `internal: true` there is no route to a resolver either. ## Where the evidence lives | Protection | Evidence | |---|---| | Non-root agent | The version's spec (`GET /v1/envs/{id}/versions`, `Spec.services[].user`, `Spec.agent`). | | Separate grader, refused files | The verify deployment's `collect`, `grade`, and `verdict` phases (`GET /v1/deployments/{id}`). | | Caps | The version's spec (`resources`) and the rendered compose on the host. | | Egress | Progress events on `cmd:` and the `prepare` phase's command result; the sidecar's `env.create` result carries `egress`. | Progress lines are stored as events with subject `cmd:`; the command id is on the `commands` row the control plane recorded for the phase. --- # Concepts Source: https://denv.ai/docs/explanation/concepts/ Section: explanation denv is the environment company. This page is the vocabulary everything else uses. ## An environment is a running system A container image is a recipe. An environment is what is running: the database with its rows, the queue with its backlog, the object store with its files, and the box an agent gets a shell in, all on one network. For a coding task that may be one container. For an ops task it is five services and a few gigabytes of state, and that is the case denv was built for. Because an environment is a system, three operations matter that a container runtime does not offer: a **snapshot** that captures every service's data at the same instant, a **fork** that gives you a new copy of the whole system with that data, and **grading apart from the agent**, so the thing that scores an episode never shares a process tree with the thing being scored. ## The four nouns | Noun | What it is | How it is created | |---|---|---| | **Project** | A repository with a kind: `rl-environment` or `agent` | `denv project create`, or connecting a GitHub repo | | **Environment** | A running stack on one target. Every project gets `alpha`. A fork is an environment whose parent is a version | deploy, fork | | **Deployment** | One change to an environment, run as ordered phases, each with evidence | deploy, snapshot, verify, fork, rollback, promote, destroy | | **Version** | Pinned image digests plus a consistent snapshot of every volume | every successful deploy or snapshot | A deployment is not a version. An environment has many deployments over its life; a version is immutable and can be forked or restored at any time. A **sandbox**, the word frameworks use, is an environment created for one episode, usually by forking a version. ## Phases and evidence Every deployment kind has a fixed list of phases. A deploy is `resolve → place → provision → join → prepare → start → health → warm → snapshot → ingress → verdict`. Each phase records evidence: which host was chosen and why, which digests were pinned, how long the writers were frozen for the snapshot, what the egress probe found. If a phase fails, the rest are skipped and the verdict reverts the environment to its previous healthy version; the deployment's status is `rolled_back`, and the agent never sees a half-deployed world. The record of phases is the thing you show someone who asks how an environment was run. It is produced by the runtime, not written by hand. ## Targets An environment runs on a target the user connects. denv never owns the compute. - **Your machines.** Any Linux box running the `denvd` sidecar, which opens one outbound connection to the control plane and never listens on a port. With a ZFS pool, snapshots are atomic across volumes and a fork takes about a second. - **Machines denv provisions in your cloud account**, on Hetzner or AWS, with the sidecar installed by cloud-init. - **Hosted sandbox providers** such as E2B and Daytona. No sidecar; the control plane drives the provider's API with your key. These run one container per environment, and snapshot and fork are as fast as the provider makes them. The console shows a capability row per target so nothing is promised a target cannot deliver. ## Protections by default For the `rl-environment` kind, every environment gets the five properties a lab's environment engineer checks for, regardless of which framework runs inside: 1. The agent is never root. A compose file whose agent service lacks a non-root `user` is refused at `resolve`. 2. The grader runs as a separate principal, after the agent's processes are stopped, on a fork. 3. Special files in submissions are refused: a symlink, FIFO, device or socket scores zero and the grader does not run. 4. Memory, process and CPU caps on every service. 5. Egress is denied, and a probe from inside the environment's network proves it before the episode. These are properties of the environment runtime, so Harbor, Inspect, OpenEnv, verifiers, HUD and Karotte all get them without changes to the task. ## The framework is yours denv does not run an agent loop, judge a task, or train a model. Harbor, Inspect, OpenEnv, verifiers and HUD do those things, and they reach denv through one small sandbox API: create, exec, files in and out, snapshot, fork, destroy. Trainers reach it through Harbor and OpenEnv. The environment is denv's; everything above it is the user's. --- # How snapshot and verify work Source: https://denv.ai/docs/explanation/snapshot-and-verify/ Section: explanation ## A consistent snapshot of a running system A snapshot of one container is easy. A snapshot of five services with state is only useful if every service's data is captured at the same instant; otherwise the fork starts with a queue that has already processed a message the database never saw. denv's snapshot does four things in order. Databases are asked to checkpoint, so their on-disk state is complete. Every container is paused, which freezes writers. Every volume is captured together: on ZFS a recursive atomic snapshot, on a plain disk a copy taken while nothing writes. Containers are unpaused and health is re-checked, because probes that ran during the freeze failed and the stack must be known good before anyone uses it. The frozen window is measured and recorded in the phase evidence. A version is that capture plus the pinned image digests. A fork creates volumes from the capture (clones on ZFS), renders the same services, and starts them on their own network. ## Why the grader runs on a fork Reinforcement learning finds whatever the reward overlooks. The attacks Karotte documented are concrete: symlink the answer file to the expected answer, replace it with a FIFO to hang the grader, leave a background process to rewrite the file after the check, replace `python3` in the path the grader uses. Every one of them works when the grader runs inside the agent's container after the agent's turn. denv's verify removes the shared surface. The agent's container is frozen first, so nothing of the agent's runs again. The declared paths are copied out with a tool that does not follow symlinks, into a directory the agent cannot reach, and the copy is checked: anything that is not a regular file or a directory, or that is too large or too deep, is refused and the episode scores zero without the grader running. The environment is snapshotted with the agent still frozen and forked. The grader runs on the fork, as root, in its own image, with the copy mounted read-only. It can still talk to the fork's database and services, because they are part of what is being graded. The fork is destroyed afterwards and the agent's environment is left as it was. The sequence is recorded phase by phase, so a reward comes with the evidence of how it was produced. ## What this costs A verify is a snapshot, a fork and a container start. On a ZFS host that is a few seconds. On a plain disk the copy dominates and it is tens of seconds. For training loops with no agent process, where a sample is just files, a lighter grade-only path is planned that skips the freeze and the snapshot. --- # Data handling Source: https://denv.ai/docs/explanation/data-handling/ Section: explanation **denv holds metadata, versions and verdicts. It never holds your environments' data.** Snapshots live on the disks of the target that took them: your machines, machines in your cloud account, or your sandbox provider's storage. The control plane records that a snapshot exists and what version it belongs to, not its contents. A fork is made on the same host as its parent snapshot. The sidecar on your machines opens one outbound connection and never listens. Commands flow from the control plane to the sidecar; the sidecar never dials a host of yours on denv's behalf. Secrets a command needs reach the host inside that command and are not stored on denv in clear. Environments, verifiers and rollouts are visible only to the team that owns them. API keys are team-scoped, can be revoked, and are stored as hashes. Sign-in is through GitHub; denv keeps your GitHub id, login, name and email. Egress is denied by default for the `rl-environment` kind and proven by a probe before each episode. Allowing egress is an explicit setting in the compose file and is visible in every deployment record. Nothing in your environments is trained on or licensed without a signed agreement. Labs that need the control plane inside their own network can run it there; it is a single container with a database. --- # Creating an environment on denv Source: https://denv.ai/blog/creating-an-environment/ Published: 2026-10-10 # Creating an environment on denv denv is the environment company: environments you can version, fork per rollout, and grade apart from the agent. Agents run in them, get evaluated in them, and models train against them. You bring the environment, built in whatever framework you already use. You connect GitHub and a place to run it. denv deploys it, keeps versions of it, snapshots it consistently, forks it for each rollout, and enforces the protections a lab will ask about. We never own your compute or your data. This post is the whole journey, end to end. The framework posts that follow assume it. ## What an environment is An environment is a running system, not a container image. For an ops task it is a database with data, a queue with a backlog, an object store, and the box the agent gets a shell in. For a coding task it may be one container. Both are environments; the first is why denv exists. denv's object model has four nouns: | Noun | What it is | Created by | |---|---|---| | **Project** | A repo with a *kind*: `rl-environment`, `agent`, `service`, `website`, `inference` | `denv project create`, or connecting a GitHub repo | | **Environment** | A running stack on one target. Every project gets `alpha` by default. A fork is an environment whose parent is a version | deploy, fork | | **Deployment** | One change to an environment, run in phases, each with evidence | deploy, snapshot, fork, rollback, promote, destroy | | **Version** | Pinned image digests plus a consistent snapshot of every volume, taken together | every successful deploy or snapshot | An environment can have many deployments over its life. A version is immutable. A sandbox, the word frameworks use, is simply an environment created for one episode, usually by forking a version. ## Step 1: describe the environment The environment is a `docker-compose.yaml` in your repo, plus a small `x-denv` block. If you already have a compose file for your framework, it is likely usable as-is. ```yaml # docker-compose.yaml x-denv: kind: rl-environment egress: deny # default; `allow` opts the stack onto the internet warm: # runs after health, before the snapshot - "psql -h db -U postgres -f /seed/orders.sql" agent: service: opsbox # where the agent gets its shell; must not run as root verifier: image: ghcr.io/you/verifier:1.2 command: ["/verify.sh"] collect: ["/home/ubuntu/answer.txt"] # copied to a root-owned location before grading clock: frozen # the stack sees the same wall clock every fork services: db: image: postgres:16-alpine environment: { POSTGRES_PASSWORD: world } volumes: ["pg:/var/lib/postgresql/data"] healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] interval: 2s retries: 30 queue: image: redis:7-alpine command: ["redis-server", "--appendonly", "yes"] volumes: ["redis:/data"] healthcheck: { test: ["CMD", "redis-cli", "ping"], interval: 2s, retries: 15 } opsbox: image: ubuntu:24.04 user: "1000:1000" command: ["sleep", "infinity"] mem_limit: 2g pids_limit: 512 environment: { HOME: /home/ubuntu, PGHOST: db, PGPASSWORD: world } volumes: ["home:/home/ubuntu"] depends_on: [db, queue] volumes: { pg: {}, redis: {}, home: {} } ``` What denv reads from this: - **Services, images, volumes, healthchecks, depends_on.** Standard compose. Every declared volume becomes a dataset that is snapshotted and forked. - **Caps.** `mem_limit`, `pids_limit`, `cpus`. Services that omit them get defaults (4 GiB, 2048 pids, 2 CPUs). Caps are rendered into the stack on every target that supports them. - **`user`.** The agent service must not be root. If `x-denv.agent.service` runs as root, the deploy fails at `resolve` with a message saying so. - **`x-denv.egress`.** `deny` by default. The stack's network has no route out, and before every episode denv runs a probe from inside the network and records that it was blocked. `allow` is explicit. - **`x-denv.warm`.** Commands run after every service is healthy and before the snapshot, so the snapshot looks like a system that has been running, not one that just booted. - **`x-denv.verifier`.** The grader. It runs as a separate principal, after the agent's processes are stopped, on a fork, with the `collect` paths copied to a root-owned location first. Symlinks, FIFOs, devices and other special files in those paths are refused and the episode scores zero. - **`x-denv.clock`.** `frozen` pins the wall clock the stack observes to the snapshot time, so a fork replays identically. `real` is the default for non-RL kinds. What denv refuses, with the reason in the deploy's `resolve` phase: `network_mode: host`, `privileged`, `cap_add`, bind mounts, `build:` without a prebuilt image on targets that cannot build, and undeclared volumes. The environment is a world, and the world should not reach the host. If you have no compose file yet, `denv init rl-environment` writes the one above. ## Step 2: connect GitHub ``` denv login denv project create --kind rl-environment --source github:you/ops-world ops-world ``` Or use the console: *New project → pick a kind → pick a repo*. denv installs a GitHub App with read access to that repo and a status check. Every push to the default branch becomes a deploy of `alpha`; branches deploy to environments named after them when you ask. The status check links to the deployment's phase log. Nothing is cloned onto denv's infrastructure beyond what the deploy needs to pin digests. Build output stays on your target. ## Step 3: connect a target A target is where environments run. You connect at least one per team; you can connect several and choose per environment. **Your own machines.** Any Linux box with Docker. Install the sidecar: ``` curl -fsSL https://get.denv.ai | sh -s -- --token $(denv host join-token) ``` `denvd` is one static binary. It opens one outbound WebSocket to the control plane and never listens on a port. The join token is single-use and becomes a per-host secret. With a ZFS pool (`--zfs-pool denv`) snapshots are atomic across every volume and a fork takes about a second; without one it reports `storage: dir` and copies, which is correct and slower. `denv host list` shows what each host reported. **Provisioned machines.** `denv cloud connect hetzner --token …` or `denv cloud connect aws --profile …`. When a deploy has no host with capacity, the `provision` phase creates one with cloud-init that installs `denvd` and joins. Machines are yours, in your account, deletable by you. **Hosted sandbox providers.** `denv cloud connect e2b --api-key …`, likewise `daytona`, `modal`, `prime`. No sidecar; the control plane drives the provider's API with your key. These targets run single-container environments only, and snapshot and fork are as fast as the provider makes them. The console shows the capability row for the target you picked, so you are never promised a feature the target cannot deliver: | Capability | Your machines (denvd) | E2B | Daytona | Modal | Prime Sandboxes | |---|---|---|---|---|---| | Multi-service stack | yes | no | no | no | compose, in progress | | Consistent snapshot across volumes | ZFS | pause + snapshot | snapshot to image | filesystem + memory snapshot | snapshot | | Fork time | ~1 s on ZFS | seconds | tens of seconds | seconds | seconds | | Egress deny, verified | yes | provider network config | limited | `block_network` | yes | | Caps enforced | yes | provider tier | provider tier | per sandbox | per sandbox | | GPU | planned | no | partial | yes | yes | ## Step 4: deploy ``` denv deploy --project ops-world --env alpha --wait ``` A deploy runs these phases, each recorded with evidence, visible in the console and the CLI as they happen: ``` resolve parse compose, apply rules, pin every image to a digest → Version place choose a host (snapshot locality, capacity) or a provider provision create a machine if the pool is empty join wait for the sidecar to connect prepare datasets created, images pulled by digest start stack up in its own network health every service passes its healthcheck warm x-denv.warm commands snapshot writers frozen, databases checkpointed, every dataset captured together ingress names and TLS for published services verdict version marked healthy; alpha points at it ``` If any phase fails, the remaining phases are skipped, the verdict records why, and the environment is restored to its previous healthy version. The deployment's status is `rolled_back`, not `failed`, and the agent never sees a half-deployed world. The snapshot is the heart of it. Databases are checkpointed, writers are paused, every volume is captured in the same instant, then writers resume and health is re-checked. The frozen window is measured and recorded in the phase evidence. A version is that snapshot plus the pinned digests. ## Step 5: use it **Fork for a rollout.** A fork is a new environment created from a version. On ZFS it is a clone, about a second, with the data present: ``` denv fork --name rollout-017 alpha ``` **Snapshot mid-episode.** Capture the live state of a running environment as a new version, so you can branch from the middle of an episode: ``` denv snapshot rollout-017 denv fork --name rollout-017-b rollout-017 ``` **Exec and files.** ``` denv exec --service opsbox rollout-017 -- psql -c "select count(*) from orders" denv files put rollout-017 --service opsbox ./fixtures /home/ubuntu/fixtures denv files get rollout-017 --service opsbox /home/ubuntu/answer.txt ./out/ ``` **Roll back.** `denv rollback alpha ver_…` restores a previous version's snapshot and re-checks health. **Verify.** `denv verify rollout-017` stops the agent's processes, collects the declared paths to a root-owned location, refuses special files, forks the environment, and runs the verifier image in the fork with those paths mounted read-only. The reward and the verifier's output land in the deployment's evidence. The agent's own fork is untouched, so you can inspect what it did. **Promote.** `denv promote ops-world alpha staging` runs the gates: fresh fork, dataset consistency, N identical replays, invariants (nothing deleted, disabled, or silenced), a soak through the next scheduled trigger, then points `staging` at the version. The gate record is the environment's data sheet. **Export.** `denv export --harbor ops-world@ver_… ./out` writes a Harbor task directory per task, with the pinned image, so the environment runs on any Harbor backend. `denv export --openenv` emits an OpenEnv provider image. ## Step 6: hand it to a framework Frameworks never see phases. They see a sandbox API: | Operation | Endpoint | |---|---| | Sandbox for one episode | `POST /v1/sandboxes` with `{from_version}` or `{image, cmd, env, ports, volumes, egress}` | | Run a command | `POST /v1/envs/{id}/exec` | | Files in and out | `PUT /v1/envs/{id}/files?service=&path=` (tar body), `GET …/files` | | Snapshot live state | `POST /v1/envs/{id}/snapshot` | | Fork | `POST /v1/envs/{id}/fork` | | Destroy | `DELETE /v1/envs/{id}` | The Python package wraps it: `pip install "denv[all]"`, then `-e denv.harbor:DenvEnvironment` for Harbor, `sandbox="denv"` for Inspect, `provider=DenvProvider()` for OpenEnv. Each framework post shows exactly how. For sandboxes created from an image, only the paths listed in `volumes` (default `/work`) persist across snapshot and fork. The image's writable layer is not data. Frameworks that write elsewhere lose it on fork, by design, and the adapters set the right paths for each framework. ## What you get by default For the `rl-environment` kind, every environment, on every target that supports it: 1. **The agent is never root.** Enforced at `resolve`. 2. **The grader is a separate principal, after the agent stops, on a fork.** The agent cannot tamper with grading because grading does not happen where the agent is. 3. **Special files are refused.** Symlinks, FIFOs, devices in collected paths score zero. 4. **Memory, pid and cpu caps on every service.** Rendered into the stack, verified by the sidecar. 5. **Egress denied and proven.** A probe runs from inside the network before every episode; the result is in the evidence. Everything the deployment did is in the phase log. That log is what you show a lab. ## Status **Ships today.** Projects with kinds (`rl-environment`, `agent`); `alpha` by default; `denvd` on your machines with ZFS or dir storage; Hetzner provisioning; deploy with the eleven phases above and rollback on failure; snapshot as a deployment kind; fork; rollback; exec; files in and out; `POST /v1/sandboxes`; caps, unprivileged agent user, egress deny with a verified probe; `x-denv.egress`, `x-denv.warm`, `x-denv.agent` (non-root enforced at resolve) and `x-denv.verifier` with `collect`; `denv verify` as a deployment kind (stop → collect → snapshot → clone → grade → verdict) with symlinks, FIFOs, devices and sockets refused for a score of 0, the grader run on a fork as root in its own image or inside a service, and the agent left frozen unless `--resume`; `denv files put|get`; Harbor, Inspect and OpenEnv adapters passing conformance, including the verify attack cases (2026-10-11). **Committed by this post.** `x-denv.clock: frozen`; promote gates and the data sheet; GitHub App and push-to-deploy; `denv login` and the console; E2B, Daytona, Modal and Prime Sandboxes targets with the capability table above; AWS provisioning; GPU device passthrough; `denv export --harbor` and `--openenv`; `denv import harbor|karotte`; host-side image builds; ingress names and TLS; MCP server. Each lands with a dated note here. --- # Harbor on denv Source: https://denv.ai/blog/harbor/ Published: 2026-10-10 # Harbor on denv Harbor is the task format labs mandate. A task is a directory: `instruction.md`, `task.toml`, an `environment/` with a Dockerfile or a compose file, `tests/test.sh` that writes `/logs/verifier/reward.txt`, and a `solution/`. Datasets are published to Harbor Hub as `org/name@tag`, and `harbor run` drives one of about forty agents against them inside an environment backend: Docker, Daytona, Modal, E2B, Runloop, GKE, and others. denv is one more backend on that list, with one difference. On every other backend an environment is a container that is rebuilt per episode. On denv it is a versioned, snapshotted, forkable system that keeps the five protections regardless of which agent runs inside. There are three ways to use them together. ## 1. denv as the Harbor backend Install the package and point Harbor at it: ``` pip install "denv[harbor]" export DENV_SERVER=https://api.denv.ai DENV_API_KEY=dak_… harbor run --dataset terminal-bench/terminal-bench-2 \ --agent claude-code \ -e denv.harbor:DenvEnvironment ``` Nothing about the dataset or the agent changes. `denv.harbor:DenvEnvironment` implements Harbor's `BaseEnvironment`: `start`, `stop`, `exec`, `upload_file`, `upload_dir`, `download_file`, `download_dir`. Here is what each does on denv. **start.** The task's `[environment]` config decides the sandbox: | In `task.toml` or `environment/` | What denv does | |---|---| | `docker_image = "…"` | `POST /v1/sandboxes {image}` with volumes for `/app`, `/logs`, and the task `workdir` | | `docker-compose.yaml` present | the file is deployed as a project under your team; the agent runs in the first service, or the one marked `x-denv.agent.service` | | only a `Dockerfile` | built on the target by the sidecar, cached by content digest, then deployed as above | | `allow_internet = true` | `egress: allow`; otherwise egress is denied and the probe result is recorded | | `cpus`, `memory_mb` | become `cpus` and `mem_limit` on the agent service; Harbor's `--cpus limit` and `--memory limit` policies are honoured, `guarantee` is reported as unsupported | The sandbox returns when every service is healthy. The directories Harbor expects, `/logs/agent`, `/logs/verifier`, `/logs/artifacts`, `/tests`, `/solution`, are created before the agent starts. **exec.** Harbor hands the adapter a shell string plus `cwd`, `env`, `user`, and a timeout. denv runs it through bash when the image has it and sh otherwise, with env exported first, as the requested user. Harbor's agents, oracle and verifier all pass through this one path. When Harbor sets a default user on the environment, that user is used; it is never root unless Harbor says `user="root"`, which it does for setup and verification. **upload and download.** Tar streams through `PUT` and `GET /v1/envs/{id}/files`. Directory download copies the directory's contents, the way `docker cp src/.` does, which is what Harbor's verifier expects when it fetches `/logs/verifier` into the trial directory. **stop.** `DELETE /v1/envs/{id}` when Harbor asks to delete; otherwise the environment is kept for inspection and reaped by TTL. **Capabilities.** The adapter reports `disable_internet=True` and `docker_compose=True`. It does not report `mounted`, so Harbor downloads logs rather than expecting a shared filesystem. GPU and TPU report false until device passthrough ships. ### What a Harbor trial gains on denv - **Every protection, without the task author doing anything.** The agent service runs as the task's user, never root; caps are enforced; egress is denied and proven before the agent's first command. Harbor's own defaults leave three of Karotte's five protections unenforced; on denv they are properties of the environment. - **Fork instead of rebuild.** Harbor rebuilds the environment for each trial. With `-k 5`, five attempts means five builds. On denv the first trial deploys a version; the next four fork it in about a second each, data included, because `DenvEnvironment` keys the version on the task digest and reuses it. - **A verdict log per trial.** Each trial's environment has a deployment record: the pinned digests, the snapshot, the egress probe result, the caps. That record is the evidence you attach to a dataset when a lab asks how it was run. - **Multi-service tasks that stay portable.** Compose tasks run on denv unchanged. If a dataset must also run on Daytona or Modal, the single-container discipline those providers need is yours to keep; denv does not force it, and `denv export --harbor` flags tasks that exceed a target's limits. ### Limits that are real Harbor's shared verifier runs `tests/test.sh` inside the same environment, as root, after the agent's turn. That is Harbor's design and the adapter honours it. If you want the grader as a separate principal on a fork, set `[verifier] environment_mode = "separate"` in `task.toml`; denv then forks the agent's environment at the end of the episode, stops the agent's processes in the fork, and runs the verifier there. The agent's environment is untouched. Both modes write the reward where Harbor expects it. ## 2. Import a Harbor task as a denv project ``` denv import harbor ./tasks/fix-backup-cron ``` Reads `task.toml` and `environment/`, writes a `docker-compose.yaml` with an `x-denv` block, creates a project of kind `rl-environment`, and deploys `alpha`. A compose task is used directly. A Dockerfile task becomes one service, built on your target. `tests/test.sh` becomes the `x-denv.verifier` command, `solution/` is kept out of the image, and the task's `allow_internet` maps to `x-denv.egress`. From there the task is a denv environment: push to redeploy, fork per rollout, snapshot mid-episode, promote through gates. You can keep publishing it to Harbor Hub from the same repo. ## 3. Export a denv environment as Harbor tasks ``` denv export --harbor ops-world@ver_01… ./dist ``` For each task declared in the project (a `tasks/` directory with `instruction.md` and a verifier, or a pipeline that generates them), writes a Harbor task directory: `task.toml` with `docker_image` pinned to the version's digest, `instruction.md`, `tests/test.sh` wrapping the verifier, `solution/` if present. Multi-service versions are exported as compose tasks, with a note of the memory and CPU a single-container target would need. Export is how a denv environment reaches every trainer that reads Harbor: rLLM, verifiers, OpenEnv's Harbor environment, SkyRL's Harbor generator. You build and calibrate on denv, you publish in the format labs consume. ## Conformance The adapter is proven by running Harbor, not by unit tests of the adapter. `make e2e-frameworks` in the backend repo boots a control plane and a sidecar on one machine and runs: ``` harbor run -p tests/fixtures/harbor/hello -a oracle -e denv.harbor:DenvEnvironment ``` The oracle agent uploads the solution, runs it, the verifier writes reward `1`, and Harbor downloads the logs. The run passes on a laptop with Docker and dir storage. We run it on every commit and will run Terminal-Bench 2 through it before calling the backend production-ready. ## Status **Ships today.** `denv.harbor:DenvEnvironment` for image-based and compose-based tasks; exec through bash/sh with env, cwd and user; file upload and download; `allow_internet` to egress; log directories pre-created; conformance with the oracle agent. **Committed by this post.** Dockerfile-only tasks built on the target (today they are refused with a message to set `docker_image`); fork reuse across `-k` attempts keyed on task digest; `cpus` and `memory_mb` mapped to caps with enforcement policies reported; `environment_mode = "separate"` running the verifier on a fork as a separate principal; `denv import harbor`; `denv export --harbor`; Terminal-Bench 2 run through the backend with published results. --- # Inspect on denv Source: https://denv.ai/blog/inspect/ Published: 2026-10-10 # Inspect on denv Inspect, from the UK AI Security Institute, is the evaluation framework with the cleanest sandbox abstraction: a `SandboxEnvironment` with `exec`, `read_file`, `write_file`, per-sample setup and cleanup, selected in a task with `sandbox="docker"` or any plugin name. Plugins register through a Python entry point, so a `pip install` is enough to add one. denv is such a plugin. ``` pip install "denv[inspect]" export DENV_SERVER=https://api.denv.ai DENV_API_KEY=dak_… ``` ## Selecting denv in a task Inspect treats a string sandbox config as a file path, so configuration lives in a file or a model, not a bare image name. ```python from inspect_ai import Task, task from inspect_ai.dataset import json_dataset from inspect_ai.scorer import includes from inspect_ai.solver import generate, use_tools from inspect_ai.tool import bash @task def ops_triage(): return Task( dataset=json_dataset("tasks.jsonl"), solver=[use_tools(bash()), generate()], scorer=includes(), sandbox=("denv", "denv.yaml"), ) ``` `denv.yaml` describes the sandbox. Two shapes: ```yaml # a single image, like Inspect's docker sandbox image: python:3.12-slim egress: deny # default volumes: [/work] # the only paths that persist across snapshot and fork ``` ```yaml # a fork of a whole denv environment: services, data, network from_version: ver_01M4KQ9F1NT5PRPYS4G85X2VCB service: opsbox # where exec runs ``` With `sandbox="denv"` and no file, the image comes from `DENV_SANDBOX_IMAGE` (default `python:3.12-slim`). A `denv.yaml` next to the task file is picked up automatically, as Inspect does for `compose.yaml`. Inline config is `DenvSandboxConfig(image=…)` or `DenvSandboxConfig(from_version=…)` from `denv.inspect`. Per-sample sandboxes work the same way: a `Sample` may carry `sandbox=("denv", "ver_…")` to fork a different version for that sample, which is how a dataset mixes worlds. ## What happens during an eval **task_init.** Credentials are checked so a missing key fails before any model call. **sample_init.** One `POST /v1/sandboxes` per sample. From an image: an environment with one service and volumes for the listed paths. From a version: a fork, data present, about a second on ZFS. The sandbox returns when healthy; the sample starts. **exec.** `sandbox().exec(cmd, cwd, env, user, timeout, input)` becomes one `POST /v1/envs/{id}/exec`. The argv is run through bash when the image has it, sh otherwise, with env exported and cwd applied. `input` is written to a file in the sandbox and redirected in. A non-root `user` is honoured with `su`. Timeouts are enforced server-side and surface as Inspect's `TimeoutError`. **write_file / read_file.** Tar through `PUT` and `GET /v1/envs/{id}/files`. Text and bytes both work. **sample_cleanup.** `DELETE /v1/envs/{id}`, unless `--no-sandbox-cleanup`, in which case the environment is kept and `inspect sandbox cleanup denv` reaps it later. Interrupted evals clean up the same way. **connection.** `inspect eval --interactive` and the `sandbox().connection()` call return `denv exec --service -- sh`, so you can attach to a sample's environment from your terminal. ## What Inspect gains on denv - **Whole-system samples.** An Inspect sandbox is normally one container, or a compose stack rebuilt per sample. With `from_version` every sample forks a running world with data in its databases. Eval of an incident-response task starts in the incident, not at boot. - **Mid-sample branching.** A solver can call `denv.Client().snapshot(env_id)` and fork from it to evaluate several continuations of the same state. Inspect has no primitive for this; denv exposes it through the client and the sample's `sandbox()` carries the environment id. - **Protections without Dockerfile work.** Caps, egress denial with a recorded probe, and a non-root agent user come from the environment. `egress: allow` in `denv.yaml` is the only way a sample reaches the internet, and the eval log shows that it did. - **An eval log that names the environment.** Each sample's events include the denv environment id and the version it was forked from. Two evals that disagree can be compared on exactly what world they ran in. ## Concurrency `default_concurrency()` is 8 sandboxes per eval; raise it with `--max-sandboxes`. Fork cost on ZFS is roughly constant, so a hundred concurrent samples is a hundred clones, not a hundred builds. On hosted sandbox targets the provider's limits apply and are shown in the console before you start. ## Scoring with a separate verifier Inspect scorers run in the eval process and call back into the sandbox. That is fine for `includes()` or a model-graded scorer. For a grader that must not share a process tree with the agent, give the task an `x-denv.verifier` in the environment's compose file and call `denv.Client().verify(env_id)` from a scorer. denv stops the agent's processes in a fork, collects the declared paths to a root-owned location, refuses special files, and runs the verifier image there. The reward comes back as the score and the verifier's evidence is in the environment's deployment record. ## Conformance `make e2e-frameworks` runs an Inspect eval with the `mockllm/model` against a denv sandbox: a solver writes a marker file, a scorer reads it back with `exec`, accuracy must be 1.0, and the sample's environment must be destroyed afterwards. The plugin is loaded through the `inspect_ai` entry point only; the test does not import `denv.inspect` directly, so a packaging mistake fails the test. ## Status **Ships today.** `sandbox="denv"`, `("denv", "denv.yaml")`, `("denv", "ver_…")`, `DenvSandboxConfig`; sandboxes from an image or a version; exec with cwd, env, user, stdin and timeout; read and write files; cleanup; connection string; entry-point registration; conformance with mockllm. **Ships today (2026-10-11).** `Client().verify()` returning the reward and the evidence, for use from a scorer. **Committed by this post.** `inspect sandbox cleanup denv`; environment id and version in the eval log's sample events; `--max-sandboxes` mapped to provider limits with the console's capability row; GPU samples once device passthrough ships. --- # OpenEnv on denv Source: https://denv.ai/blog/openenv/ Published: 2026-10-10 # OpenEnv on denv OpenEnv is the wire protocol for RL environments: `reset`, `step`, `state` over HTTP and WebSocket, tools over MCP, one Docker image per environment, distributed on Hugging Face Spaces. Its committee spans Meta, Hugging Face, Nvidia, Mercor, Fleet and Microsoft, and eight-plus trainers consume it directly, including TRL, SkyRL, torchforge and Unsloth. OpenEnv's own `harbor` environment wraps any Harbor dataset, which is how most coding tasks reach it. Where an OpenEnv environment runs is a *provider*: `LocalDockerProvider`, `DockerSwarmProvider`, `KubernetesProvider`, or a hosted runtime. OpenEnv's RFC 002 leaves suspend, resume and snapshot "provider-local". denv is the provider that makes them real. ``` pip install "denv[openenv]" export DENV_SERVER=https://api.denv.ai DENV_API_KEY=dak_… ``` ## Running an OpenEnv image on denv ```python from denv.openenv import DenvProvider from my_env import MyEnv # an OpenEnv client class env = MyEnv.from_docker_image("ghcr.io/you/my-env:v1", provider=DenvProvider()) obs = env.reset() obs = env.step(action) env.close() ``` `DenvProvider` implements OpenEnv's `ContainerProvider`: - **start_container(image, port, env_vars).** `POST /v1/sandboxes {image, ports:[port], env}`. The environment's HTTP port is published and the base URL returned. The provider waits for the environment to be healthy before returning it. - **wait_for_ready(base_url).** Polls the OpenEnv `/health` route. - **stop_container / close.** `DELETE /v1/envs/{id}`. Published ports require `egress: allow` today, because a denied network is `internal` and cannot publish. `DenvProvider(egress="allow")` makes that explicit. When the ingress proxy ships, a denied environment can still be reached on its published port through denv, and `deny` becomes the default here too. ## Forking an episode This is the part no other provider has: ```python root = MyEnv.from_docker_image("ghcr.io/you/my-env:v1", provider=DenvProvider()) root.reset() for a in prefix: root.step(a) branches = [root.provider.fork() for _ in range(8)] ``` `fork()` snapshots the environment's live state as a new version, then creates a new environment from it, and returns a provider bound to the child with its own base URL. Each branch continues from the same state; none shares memory with the root. For tree search, best-of-N from a common prefix, or counterfactual evaluation, this replaces replaying the prefix eight times. What persists across the fork is what the environment keeps in its `volumes` (default `/work`) and in any declared service volumes. Process memory does not survive; the OpenEnv server restarts in the fork and must rebuild its in-memory state from disk on startup, which well-behaved OpenEnv environments already do for `reset` idempotence. The child's `state()` after fork should equal the parent's; the conformance suite checks this for the reference environments. ## Multi-service OpenEnv environments An OpenEnv image is one container, but an OpenEnv environment often needs a database or a browser next to it. On denv, put the OpenEnv server in a compose file with its dependencies and `x-denv.agent.service` pointing at it; deploy it as a project; then create sandboxes from the version: ```python env = MyEnv.from_denv_version("ver_01M4…", provider=DenvProvider()) ``` `from_denv_version` is the one addition to the client surface, provided by `denv.openenv`. It forks the version and connects to the OpenEnv server's published port. The database, the queue and the seeded data are in the fork. ## Exporting a denv version as an OpenEnv environment ``` denv export --openenv ops-world@ver_01… ./dist ``` Writes an OpenEnv environment package: a server that exposes `reset`, `step`, `state` and the declared MCP tools, and a `Dockerfile` whose base is the version's agent service image, pinned by digest. `reset` forks the version through denv rather than restarting a container, so an exported environment keeps denv's semantics when it runs under a trainer that only knows OpenEnv. Credentials are read from `DENV_API_KEY` in the trainer's environment. For environments that are a Harbor dataset, prefer `denv export --harbor` and OpenEnv's own `harbor` wrapper; it already handles sixteen validated agents and token capture. ## Token and logprob capture OpenEnv's capture proxy returns `prompt_token_ids`, `completion_token_ids` and per-token logprobs from vLLM or SGLang so a trainer can do policy gradient on an agent's actual tokens. denv does not touch the model side. The proxy runs where the trainer runs; denv runs the environment. When the agent inside the environment calls a model, point it at the proxy as you would with any provider. ## Conformance `make e2e-frameworks` starts `python -m http.server` as a stand-in OpenEnv image through `DenvProvider`, waits for ready, and stops it. The committed conformance run adds OpenEnv's reference `echo` environment: reset, step, fork, compare `state()` across root and branch, close. ## Status **Ships today.** `DenvProvider` with `start_container`, `wait_for_ready`, `stop_container`, `close`; `fork()` returning a child provider; conformance with a stand-in image. **Committed by this post.** `egress: deny` with published ports through the ingress proxy; `from_denv_version`; `denv export --openenv`; conformance against OpenEnv's reference environments with `state()` equality across fork; listing of denv as a provider in OpenEnv's documentation. --- # verifiers (Prime Intellect) on denv Source: https://denv.ai/blog/verifiers/ Published: 2026-10-10 # verifiers on denv verifiers is Prime Intellect's environment library and the input format for prime-rl. An environment is a Python package, `vf-`, exposing `load_environment()` that returns a `vf.Environment`: `SingleTurnEnv`, `MultiTurnEnv`, `ToolEnv`, `StatefulToolEnv`, or `SandboxEnv`, each with a dataset, a parser and a rubric that produces rewards. `vf-eval` runs an environment against any OpenAI-compatible model; prime-rl trains against it. The Environments Hub hosts hundreds. verifiers v1 added a built-in **Harbor taskset**: hand it a Harbor dataset and the reward is inherited from the dataset's verifier, with no reward function to write. That is the shortest path to training on a denv environment, so it comes first. ## Path 1: Harbor taskset, denv as the backend Export the environment as Harbor tasks and point the taskset at them: ``` denv export --harbor ops-world@ver_01… ./dist/ops-world harbor publish ./dist/ops-world --tag v1 # optional; a local path works too ``` ```python import verifiers as vf env = vf.load_environment( "harbor", dataset="you/ops-world@v1", agent="claude-code", environment="denv.harbor:DenvEnvironment", # Harbor's -e flag, passed through ) ``` Every rollout is a Harbor trial with denv as the sandbox: the task's environment is forked from the exported version in about a second, the agent runs inside with caps and no egress, the verifier writes the reward, and verifiers reads it. prime-rl sees a normal taskset. The denv deployment record for each trial is available by environment id when you need to show how a reward was produced. Prime's own sandboxes and Modal are the other backends the taskset supports. Choose denv when the task is more than one container, when you want fork instead of rebuild across `n` rollouts of the same prompt, or when the environment must run inside your own cloud. ## Path 2: a denv sandbox inside a verifiers environment For a `ToolEnv` or `SandboxEnv` whose tools execute commands, denv provides the sandbox the tools run in. ```python import verifiers as vf from denv import Client class OpsEnv(vf.StatefulToolEnv): def setup_state(self, state, **kwargs): c = Client() sbx = c.sandbox(from_version=self.version) # a fork of the world state["denv"] = sbx.id return state async def bash(self, cmd: str, state) -> str: """Run a shell command in the environment.""" r = Client().exec(state["denv"], "opsbox", ["bash", "-lc", cmd], timeout_sec=120) return r.stdout + r.stderr async def cleanup(self, state): Client().destroy(state["denv"]) def load_environment(version: str = "ver_01…", **kwargs): env = OpsEnv(dataset=load_dataset("you/ops-prompts"), rubric=OpsRubric(), **kwargs) env.version = version env.add_tool(env.bash, args_to_skip=["state"]) return env ``` verifiers' `SandboxEnv` ships with Prime's sandbox client. `denv.verifiers.DenvSandboxClient` matches that client's methods, `create`, `exec`, `upload`, `download`, `destroy`, so a `SandboxEnv` can switch sandbox with one constructor argument rather than the subclass above: ```python env = vf.SandboxEnv(sandbox_client=DenvSandboxClient(from_version="ver_01…"), ...) ``` ## Rewards with a separate grader A verifiers rubric is Python in the trainer's process and may call back into the sandbox. For a grader that must not share a process tree with the agent, add a reward function that calls `Client().verify(env_id)`. denv stops the agent in a fork, collects the declared paths to a root-owned location, refuses special files, runs the `x-denv.verifier` image, and returns the reward. The rubric adds it like any other reward function, weighted as you choose. The agent's own fork stays as it was for inspection. ## Rollouts at training scale prime-rl runs many rollouts of the same prompt concurrently. With a denv version as the root, each is a fork, so the cost of `n=16` is sixteen clones rather than sixteen builds or sixteen image pulls. Fork time on ZFS is roughly constant with data size; on a hosted sandbox target it is the provider's snapshot-restore time, shown in the console before training starts. Rollout sandboxes have a TTL (default 2 hours) and are destroyed when `cleanup` runs or the TTL passes, whichever is first, so a crashed trainer does not leave a thousand environments running in your account. ## Token capture verifiers' interception server records the agent's tokens and logprobs from the model endpoint; denv does not take part in that. The environment and the model are on different sides of the trainer, as they should be. ## Conformance The committed conformance run installs `verifiers` and `prime-rl` in the suite's virtualenv, runs `vf-eval` on the Harbor taskset pointed at the `hello` fixture task with `-e denv.harbor:DenvEnvironment`, and asserts a reward of 1 from the oracle; then runs a `SandboxEnv` with `DenvSandboxClient` through one `exec`. ## Status **Ships today.** Path 1 works with any released verifiers that has the Harbor taskset, since the denv side is the Harbor adapter, which is shipped. The raw `denv.Client` calls in Path 2 work. **Ships today (2026-10-11).** `Client().verify()`: agent frozen, submission collected to a root-owned copy with special files refused, graded on a fork. **Committed by this post.** `denv.verifiers.DenvSandboxClient` matching Prime's sandbox client surface; rollout TTLs; `vf-eval` and `SandboxEnv` in the conformance suite; a published `vf-denv-ops` example environment on the Environments Hub. --- # HUD on denv Source: https://denv.ai/blog/hud/ Published: 2026-10-10 # HUD on denv HUD builds environments as Docker images with an MCP controller inside: the agent speaks MCP to the environment, the environment exposes tools such as a shell, a browser, or a desktop, and a task is an async function that yields the prompt and then the reward. `hud eval` runs a dataset; `hud dev` iterates on an environment locally. Runtimes decide where the image runs: local, Docker, Modal, Daytona, a TCP attach to something already running, or HUD's hosted runtime. HUD interoperates with Harbor datasets and trains through Tinker. denv is a runtime for HUD, and the way to run HUD environments inside your own cloud. ``` pip install "denv[hud]" export DENV_SERVER=https://api.denv.ai DENV_API_KEY=dak_… ``` ## denv as the HUD runtime ```python import hud from denv.hud import DenvRuntime env = hud.Environment("ghcr.io/you/ops-env:v2", runtime=DenvRuntime()) @env.template() async def triage(prompt: str): yield prompt yield await env.grade() # 0–1 ``` `DenvRuntime` implements HUD's runtime protocol: start the image as a denv sandbox, expose the controller's MCP endpoint, exec for setup steps, stop. Concretely: - **start.** `POST /v1/sandboxes {image, ports:[controller_port], egress}`. The MCP controller's port is published and HUD connects to it. The sandbox returns when the controller answers. - **MCP transport.** HTTP over the published port today. Over stdio through `denv exec` when the environment prefers it, which also works with egress denied. - **exec.** HUD's `setup` and `evaluate` steps run through `POST /v1/envs/{id}/exec`. - **stop.** `DELETE /v1/envs/{id}`. HUD's `HUDRuntime` rejects GPU requests and resource limits. `DenvRuntime` accepts `cpus`, `memory_mb` and `pids` and renders them as caps; GPU follows device passthrough. ## A forkable world under HUD HUD rebuilds the environment per task. On denv, a task can fork a version instead: ```python env = hud.Environment.from_denv_version("ver_01M4…", runtime=DenvRuntime()) ``` The fork carries the services and data declared in the environment's compose file; the HUD controller runs in the `x-denv.agent.service`. Browser tasks against a seeded SaaS replica, or an ops task that starts in the middle of an incident, begin in that state rather than booting into it. Fork cost is about a second on ZFS and does not grow with the size of the data. `env.fork()` during a task snapshots the live environment and returns a new `Environment` bound to the child, so a grader can evaluate several continuations from one state, or a task can checkpoint before a risky step. ## Grading as a separate principal HUD graders are inline Python, Bash in the environment, an LLM judge, or a combination, and HUD supports a separate verifier environment. On denv the separate verifier is the default for the `rl-environment` kind: `env.grade()` calls `denv verify`, which stops the agent's processes in a fork, collects the paths declared in `x-denv.verifier.collect` to a root-owned location, refuses special files, and runs the verifier image there. The reward is the task's second yield; the verifier's evidence is in the environment's deployment record. The agent's environment is untouched for inspection. Inline and LLM-judge graders work unchanged; they simply call back into the sandbox or the model. ## Datasets between HUD and Harbor HUD already converts Harbor tasks. denv adds the other direction and keeps the environment constant across formats: `denv export --harbor` from the same version that `DenvRuntime` runs under HUD, so a dataset published on both HUD and Harbor Hub refers to the same pinned images and the same verifier. ## Hosted targets HUD's own Modal and Daytona runtimes run the environment on those providers. When denv's E2B, Daytona, Modal and Prime targets ship, `DenvRuntime` can place the sandbox on any of them while keeping denv's versions, verdict log and verifier semantics; the console shows the capability row for the chosen target. Until then `DenvRuntime` places on your own machines. ## Conformance The committed conformance run installs `hud-python`, starts HUD's reference `text` environment through `DenvRuntime`, runs one task through `hud eval` with a mock model, asserts a reward is produced, and checks the environment is destroyed. ## Status **Ships today.** Running a HUD environment image as a denv sandbox with the raw client, and attaching HUD to its published MCP port through HUD's TCP runtime. The `denv.hud` package is not yet published. **Ships today (2026-10-11).** `denv verify` itself, callable from any grader through `Client().verify()`. **Committed by this post.** `denv.hud.DenvRuntime` with HTTP and stdio MCP transports; `from_denv_version` and `env.fork()`; `env.grade()` wired to it; caps from HUD resource requests; hosted targets through the same runtime; conformance with HUD's reference environment. --- # Karotte on denv Source: https://denv.ai/blog/karotte/ Published: 2026-10-10 # Karotte on denv Karotte, from Preference Model, is the harness that made protection the default. An environment is one self-contained image; a task is a Python `Step` with `instructions`, `submission_paths` and a `judge`; `karotte run` boots the image in a Firecracker microVM on Linux or an Apple container on macOS, runs the student unprivileged, kills every student process, copies the submission to a root-owned location, refuses symlinks and other special files, and only then runs the judge. Memory, process and disk caps are enforced by the VM. The network is off and checked before the run. Karotte published a table comparing those five protections across Harbor, HUD, AgentEnv, verifiers and Inspect. We agree with the table. denv's position is that the protections belong to the environment runtime, so every framework gets them, and that an environment also needs the things Karotte deliberately leaves out: more than one service, data with versions, a consistent snapshot, a fork, and a place to run that is your own cloud. There are two ways to use them together. ## 1. Run Karotte inside a denv environment A Karotte environment image is a service in a denv compose file. denv supplies the outer protections; Karotte supplies the task semantics. ```yaml x-denv: kind: rl-environment egress: deny agent: service: karotte verifier: service: karotte command: ["karotte", "judge", "--step", "fix_backup"] collect: ["/home/student/submission"] services: karotte: image: ghcr.io/you/karotte-ops-env:3 user: "1000:1000" command: ["sleep", "infinity"] mem_limit: 4g pids_limit: 1024 volumes: ["student:/home/student"] db: image: postgres:16-alpine volumes: ["pg:/var/lib/postgresql/data"] healthcheck: { test: ["CMD-SHELL", "pg_isready -U postgres"], interval: 2s, retries: 30 } volumes: { student: {}, pg: {} } ``` Deploy it, fork it per episode, and run the student with `denv exec --service karotte … -- karotte run --step fix_backup`. The judge runs through `denv verify`: the student's processes are stopped in a fork, `/home/student/submission` is collected to a root-owned location, special files are refused and scored zero, and the judge command runs there. That is Karotte's sequence, executed by denv, with the database alongside. What changes relative to `karotte run` on a laptop: | | Karotte alone | Karotte inside denv | |---|---|---| | Isolation | Firecracker (Linux) or Apple container; Docker fallback without memory limits | Container with caps on your machines today; gVisor runtime on hosts that have it; Firecracker on targets that offer it | | Services | one image | any number, with healthchecks and data | | Snapshot and fork | none | consistent snapshot of every volume; fork in about a second on ZFS | | Versions | image tag | pinned digests plus snapshot, with a deployment record per change | | Where it runs | your laptop or Linux box | your cloud, provisioned or connected; hosted sandbox targets later | | Egress | off, checked | off, probed before every episode, result in the record | The isolation row is the honest one. Karotte alone gives you a microVM. denv on your own machines gives you a container with caps by default, a gVisor runtime where the host has it, and a microVM only on targets that provide one. If VM isolation is a requirement, deploy to a target that offers it, and the console's capability row says which. ## 2. Import a Karotte environment as a denv project ``` denv import karotte ./ops-env ``` Reads the Karotte environment directory, uses the built image (or builds it on the target), and writes a compose file like the one above with an `x-denv` block derived from the Karotte steps: the student user becomes `user`, `submission_paths` become `verifier.collect`, the judge becomes the verifier command, and each `Step` becomes a task in the project's `tasks/` directory. The project deploys as `alpha` and from there is a denv environment: push to redeploy, fork per rollout, promote through gates, export as Harbor tasks. The Karotte source stays authoritative for the task logic. denv does not rewrite steps or judges; it runs them in a world. ## What denv adds that Karotte does not want to Karotte's authors are explicit that the framework is one image, no cloud integrations, no snapshots, binary egress. That is a good harness. The things an environment program needs beyond a harness are what denv is for: - **A version with a snapshot.** Two runs disagree; which world were they in? A denv version answers that with digests and a captured dataset. - **Fork mid-episode.** Branch an episode eight ways from one state for search or counterfactual grading. - **Promote gates.** A new version of the environment passes replay, consistency, invariant and soak gates before `staging` points at it. The gate record is the data sheet a lab asks for. - **Your cloud.** The environment runs in your account, on machines you can delete, with denv holding only the control plane. ## Conformance The committed conformance run builds Karotte's smallest example environment, imports it with `denv import karotte`, forks the deployed version, runs the student step with `karotte run` through `denv exec`, and grades through `denv verify`, asserting the judge's result equals Karotte's own result on the same image. A second case places a symlink in the submission path and asserts a zero. ## Status **Ships today.** Running a Karotte image as a service with denv's caps, non-root user and verified egress denial; fork and snapshot of the environment; `denv verify` with `x-denv.verifier.service` running the judge inside the fork as root over the collected copy, special files refused (2026-10-11). **Committed by this post.** `denv import karotte`; gVisor runtime selection per host; microVM targets with the capability row; the two conformance cases above; our own column in Karotte's protections table, published with the evidence. --- # Training against denv environments Source: https://denv.ai/blog/trainers/ Published: 2026-10-10 # Training against denv environments Trainers do not have an environment abstraction in common. SkyRL's `BaseTextEnv` is synchronous and in-process. verl has `AgentLoop`. prime-rl consumes verifiers environments. TRL consumes OpenEnv. Tinker is a training API, so HUD and SkyRL wrap it. rLLM reads Harbor and emits to verl, Tinker and Fireworks. What they share is two inputs: **Harbor task directories** and **OpenEnv environments**. Both already exist in each trainer's ecosystem, and both are formats denv exports and runs. denv does not ship a trainer adapter, because the plumbing exists and is maintained by people whose job it is. The environment is denv's. ## The map | Trainer | Reads | Reach denv through | Sandbox backend choice | |---|---|---|---| | **SkyRL** | Harbor (generator), OpenEnv (wrapper), skyrl-agent | `denv export --harbor` with `-e denv.harbor:DenvEnvironment`; or `DenvProvider` under the OpenEnv wrapper | Harbor's `-e` flag | | **verl** | rLLM, OpenEnv | rLLM on an exported Harbor dataset; or OpenEnv with `DenvProvider` | rLLM's backend setting | | **rLLM** | Harbor | `denv export --harbor`, backend `denv` | `--env-backend` | | **prime-rl** | verifiers | verifiers' Harbor taskset with `-e denv.harbor:DenvEnvironment`; or `DenvSandboxClient` in a `SandboxEnv` | taskset `environment` argument | | **TRL** | OpenEnv | `MyEnv.from_docker_image(..., provider=DenvProvider())` or an exported OpenEnv environment | the provider | | **Tinker** | via HUD or SkyRL | HUD on `DenvRuntime`; SkyRL as above | the wrapper's setting | | **Unsloth, torchforge, Miles** | OpenEnv | as TRL | the provider | Every row ends in one of two adapters, Harbor's or OpenEnv's, which the earlier posts describe. If a trainer appears that reads neither, it will read one of them within a release; that is where the field has converged. ## What changes for a training run **Rollouts fork.** A GRPO group of 16 rollouts on one prompt is 16 forks of one version, each with the data present, each about a second on ZFS. On a rebuilt-per-episode backend the same group is 16 builds or pulls, and the trainer's GPUs wait for them. Cold-start variance across sandbox substrates has been measured at over 100×; a fork's cost is roughly constant. **Rewards come from a separate principal.** With `x-denv.verifier` declared, the reward for each rollout is produced by `denv verify`: the agent stopped in a fork, submissions collected to a root-owned location, special files refused, grader run there. Reward hacking that works by editing the grader or the answer key has nowhere to act. The verifier's output for every rollout is in the deployment record, so a reward you do not believe can be traced. **The environment has a version.** A training run records the version it trained against. When the environment changes, a new version is promoted through gates (fresh fork, consistency, N identical replays, invariants, soak) before any trainer sees it. A reward curve that moves when the environment moved is distinguishable from one that moved because the policy did. **Egress is denied and proven.** The agent inside the environment cannot look up the answer, and the probe result for every rollout says so. The model endpoint the agent calls is reached through the trainer's capture proxy or interception server, which sits outside the environment; `egress: allow` is only for the rare environment that must reach the internet, and the record shows it. **Concurrency is a target property.** Hundreds of concurrent forks on a bare-metal ZFS host are ordinary. On a hosted sandbox target the provider's concurrency limit applies and is shown in the console before the run starts. ## Token and logprob capture Policy-gradient training needs the agent's actual prompt and completion token ids and the sampling logprobs. OpenEnv's capture proxy, verifiers' interception server and rLLM's gateway all do this at the model endpoint, usually vLLM with token ids returned and processed logprobs. denv runs the environment and does not sit between the agent and the model. Point the agent inside the environment at the trainer's proxy, which is on the allowed side of the network boundary because the proxy connects in, or run the agent harness outside the environment and let it call `exec` in. ## A concrete loop: prime-rl on an ops world ``` denv deploy --project ops-world --wait # version ver_A denv promote ops-world alpha staging # gates pass; staging → ver_A denv export --harbor ops-world@ver_A ./dist/ops-world harbor publish ./dist/ops-world --tag v1 ``` ```python env = vf.load_environment("harbor", dataset="you/ops-world@v1", agent="claude-code", environment="denv.harbor:DenvEnvironment") ``` prime-rl rolls out; each rollout forks `ver_A`, runs the agent with caps and no egress, grades with the task's verifier, and reports a reward. A week later you fix a seed bug; `denv deploy` produces `ver_B`; `promote` runs the gates; export and publish `v2`; the training config points at `v2` and the change is in the run's record. ## Conformance The committed conformance run trains nothing; trainers are heavy and their correctness is theirs. It verifies the two inputs: an exported Harbor dataset runs under Harbor with the denv backend and under OpenEnv's `harbor` environment with the denv backend selected, and an exported OpenEnv environment passes OpenEnv's reference client checks. Each trainer's own example then runs unchanged, and we publish one such run per trainer, with the denv version recorded, as documentation rather than as tests. ## Status **Ships today.** Harbor backend and OpenEnv provider, which is what every row of the map reaches. A trainer that already consumes Harbor datasets with a `-e` setting, or OpenEnv environments with a provider, can train against a denv environment now. **Ships today (2026-10-11).** `denv verify` as the reward path. **Committed by this post.** `denv export --harbor` and `--openenv`; promote gates; rollout TTLs; one published example run per trainer in the map with its denv version; a `denv` backend entry in rLLM's and OpenEnv's backend lists.