Blog · 07 · 2026-10-10 · 4 min

Training against denv environments

SkyRL, verl, rLLM, prime-rl, TRL and Tinker already read Harbor and OpenEnv. denv builds and runs the environment; the trainer reaches it through those two formats, and rollouts fork instead of rebuild.

Trainers do not have an environment abstraction in common. SkyRL's BaseTextEnv is synchronous and in-process. verl has AgentLoop. prime-rl consumes verifiers environments. TRL consumes OpenEnv. Tinker is a training API, so HUD and SkyRL wrap it. rLLM reads Harbor and emits to verl, Tinker and Fireworks.

What they share is two inputs: Harbor task directories and OpenEnv environments. Both already exist in each trainer's ecosystem, and both are formats denv exports and runs. denv does not ship a trainer adapter, because the plumbing exists and is maintained by people whose job it is. The environment is denv's.

The map

Frameworks on the left call one sandbox API: create, exec, files, snapshot, fork, destroy. denv implements it and places the environment on the target the user connected: their own machines, provisioned clouds, or hosted sandbox providers.FRAMEWORKSHarborInspectOpenEnvverifiersHUDKarotteSkyRL · verl · TRL · prime-rlDENV · SANDBOX APIcreateexecfilessnapshot · whole systemfork · whole systemdestroyversions · verdict logTARGETSyour machines · denvdHetzner · AWS (provisioned)E2BDaytonaModalPrime Sandboxes
Fig. 1 denv sits between the framework and the target. Frameworks only see the sandbox API.
Trainer Reads Reach denv through Sandbox backend choice
SkyRL Harbor (generator), OpenEnv (wrapper), skyrl-agent denv export --harbor with -e denv.harbor:DenvEnvironment; or DenvProvider under the OpenEnv wrapper Harbor's -e flag
verl rLLM, OpenEnv rLLM on an exported Harbor dataset; or OpenEnv with DenvProvider rLLM's backend setting
rLLM Harbor denv export --harbor, backend denv --env-backend
prime-rl verifiers verifiers' Harbor taskset with -e denv.harbor:DenvEnvironment; or DenvSandboxClient in a SandboxEnv taskset environment argument
TRL OpenEnv MyEnv.from_docker_image(..., provider=DenvProvider()) or an exported OpenEnv environment the provider
Tinker via HUD or SkyRL HUD on DenvRuntime; SkyRL as above the wrapper's setting
Unsloth, torchforge, Miles OpenEnv as TRL the provider

Every row ends in one of two adapters, Harbor's or OpenEnv's, which the earlier posts describe. If a trainer appears that reads neither, it will read one of them within a release; that is where the field has converged.

What changes for a training run

Version A is forked into an episode. Midway, the live state is snapshotted as version B, and eight branches fork from B in about a second each, data included. The original episode continues untouched.VERSION A · HEALTHY SNAPSHOTver_Afork ≈ 1 sstep 1step 2snapshot → ver_Bstep 3 …8 branches from ver_Bgraded ✓discardedonly declared volumes persist across snapshot and fork · process memory does not
Fig. 2 Snapshot mid-episode, then fork. Branches share state at the snapshot and nothing after.

Rollouts fork. A GRPO group of 16 rollouts on one prompt is 16 forks of one version, each with the data present, each about a second on ZFS. On a rebuilt-per-episode backend the same group is 16 builds or pulls, and the trainer's GPUs wait for them. Cold-start variance across sandbox substrates has been measured at over 100×; a fork's cost is roughly constant.

Rewards come from a separate principal. With x-denv.verifier declared, the reward for each rollout is produced by denv verify: the agent stopped in a fork, submissions collected to a root-owned location, special files refused, grader run there. Reward hacking that works by editing the grader or the answer key has nowhere to act. The verifier's output for every rollout is in the deployment record, so a reward you do not believe can be traced.

The environment has a version. A training run records the version it trained against. When the environment changes, a new version is promoted through gates (fresh fork, consistency, N identical replays, invariants, soak) before any trainer sees it. A reward curve that moves when the environment moved is distinguishable from one that moved because the policy did.

Egress is denied and proven. The agent inside the environment cannot look up the answer, and the probe result for every rollout says so. The model endpoint the agent calls is reached through the trainer's capture proxy or interception server, which sits outside the environment; egress: allow is only for the rare environment that must reach the internet, and the record shows it.

Concurrency is a target property. Hundreds of concurrent forks on a bare-metal ZFS host are ordinary. On a hosted sandbox target the provider's concurrency limit applies and is shown in the console before the run starts.

Token and logprob capture

Policy-gradient training needs the agent's actual prompt and completion token ids and the sampling logprobs. OpenEnv's capture proxy, verifiers' interception server and rLLM's gateway all do this at the model endpoint, usually vLLM with token ids returned and processed logprobs. denv runs the environment and does not sit between the agent and the model. Point the agent inside the environment at the trainer's proxy, which is on the allowed side of the network boundary because the proxy connects in, or run the agent harness outside the environment and let it call exec in.

A concrete loop: prime-rl on an ops world

denv deploy --project ops-world --wait                  # version ver_A
denv promote ops-world alpha staging                    # gates pass; staging → ver_A
denv export --harbor ops-world@ver_A ./dist/ops-world
harbor publish ./dist/ops-world --tag v1
env = vf.load_environment("harbor", dataset="you/ops-world@v1",
                          agent="claude-code",
                          environment="denv.harbor:DenvEnvironment")

prime-rl rolls out; each rollout forks ver_A, runs the agent with caps and no egress, grades with the task's verifier, and reports a reward. A week later you fix a seed bug; denv deploy produces ver_B; promote runs the gates; export and publish v2; the training config points at v2 and the change is in the run's record.

Conformance

The committed conformance run trains nothing; trainers are heavy and their correctness is theirs. It verifies the two inputs: an exported Harbor dataset runs under Harbor with the denv backend and under OpenEnv's harbor environment with the denv backend selected, and an exported OpenEnv environment passes OpenEnv's reference client checks. Each trainer's own example then runs unchanged, and we publish one such run per trainer, with the denv version recorded, as documentation rather than as tests.

Status

Ships today. Harbor backend and OpenEnv provider, which is what every row of the map reaches. A trainer that already consumes Harbor datasets with a -e setting, or OpenEnv environments with a provider, can train against a denv environment now.

Ships today (2026-10-11). denv verify as the reward path.

Committed by this post. denv export --harbor and --openenv; promote gates; rollout TTLs; one published example run per trainer in the map with its denv version; a denv backend entry in rLLM's and OpenEnv's backend lists.