Blog · 07 · 2026-10-10 · 4 min
Training against denv environments
SkyRL, verl, rLLM, prime-rl, TRL and Tinker already read Harbor and OpenEnv. denv builds and runs the environment; the trainer reaches it through those two formats, and rollouts fork instead of rebuild.
Trainers do not have an environment abstraction in common. SkyRL's BaseTextEnv is synchronous and in-process. verl has AgentLoop. prime-rl consumes verifiers environments. TRL consumes OpenEnv. Tinker is a training API, so HUD and SkyRL wrap it. rLLM reads Harbor and emits to verl, Tinker and Fireworks.
What they share is two inputs: Harbor task directories and OpenEnv environments. Both already exist in each trainer's ecosystem, and both are formats denv exports and runs. denv does not ship a trainer adapter, because the plumbing exists and is maintained by people whose job it is. The environment is denv's.
The map
| Trainer | Reads | Reach denv through | Sandbox backend choice |
|---|---|---|---|
| SkyRL | Harbor (generator), OpenEnv (wrapper), skyrl-agent | denv export --harbor with -e denv.harbor:DenvEnvironment; or DenvProvider under the OpenEnv wrapper |
Harbor's -e flag |
| verl | rLLM, OpenEnv | rLLM on an exported Harbor dataset; or OpenEnv with DenvProvider |
rLLM's backend setting |
| rLLM | Harbor | denv export --harbor, backend denv |
--env-backend |
| prime-rl | verifiers | verifiers' Harbor taskset with -e denv.harbor:DenvEnvironment; or DenvSandboxClient in a SandboxEnv |
taskset environment argument |
| TRL | OpenEnv | MyEnv.from_docker_image(..., provider=DenvProvider()) or an exported OpenEnv environment |
the provider |
| Tinker | via HUD or SkyRL | HUD on DenvRuntime; SkyRL as above |
the wrapper's setting |
| Unsloth, torchforge, Miles | OpenEnv | as TRL | the provider |
Every row ends in one of two adapters, Harbor's or OpenEnv's, which the earlier posts describe. If a trainer appears that reads neither, it will read one of them within a release; that is where the field has converged.
What changes for a training run
Rollouts fork. A GRPO group of 16 rollouts on one prompt is 16 forks of one version, each with the data present, each about a second on ZFS. On a rebuilt-per-episode backend the same group is 16 builds or pulls, and the trainer's GPUs wait for them. Cold-start variance across sandbox substrates has been measured at over 100×; a fork's cost is roughly constant.
Rewards come from a separate principal. With x-denv.verifier declared, the reward for each rollout is produced by denv verify: the agent stopped in a fork, submissions collected to a root-owned location, special files refused, grader run there. Reward hacking that works by editing the grader or the answer key has nowhere to act. The verifier's output for every rollout is in the deployment record, so a reward you do not believe can be traced.
The environment has a version. A training run records the version it trained against. When the environment changes, a new version is promoted through gates (fresh fork, consistency, N identical replays, invariants, soak) before any trainer sees it. A reward curve that moves when the environment moved is distinguishable from one that moved because the policy did.
Egress is denied and proven. The agent inside the environment cannot look up the answer, and the probe result for every rollout says so. The model endpoint the agent calls is reached through the trainer's capture proxy or interception server, which sits outside the environment; egress: allow is only for the rare environment that must reach the internet, and the record shows it.
Concurrency is a target property. Hundreds of concurrent forks on a bare-metal ZFS host are ordinary. On a hosted sandbox target the provider's concurrency limit applies and is shown in the console before the run starts.
Token and logprob capture
Policy-gradient training needs the agent's actual prompt and completion token ids and the sampling logprobs. OpenEnv's capture proxy, verifiers' interception server and rLLM's gateway all do this at the model endpoint, usually vLLM with token ids returned and processed logprobs. denv runs the environment and does not sit between the agent and the model. Point the agent inside the environment at the trainer's proxy, which is on the allowed side of the network boundary because the proxy connects in, or run the agent harness outside the environment and let it call exec in.
A concrete loop: prime-rl on an ops world
denv deploy --project ops-world --wait # version ver_A
denv promote ops-world alpha staging # gates pass; staging → ver_A
denv export --harbor ops-world@ver_A ./dist/ops-world
harbor publish ./dist/ops-world --tag v1
env = vf.load_environment("harbor", dataset="you/ops-world@v1",
agent="claude-code",
environment="denv.harbor:DenvEnvironment")
prime-rl rolls out; each rollout forks ver_A, runs the agent with caps and no egress, grades with the task's verifier, and reports a reward. A week later you fix a seed bug; denv deploy produces ver_B; promote runs the gates; export and publish v2; the training config points at v2 and the change is in the run's record.
Conformance
The committed conformance run trains nothing; trainers are heavy and their correctness is theirs. It verifies the two inputs: an exported Harbor dataset runs under Harbor with the denv backend and under OpenEnv's harbor environment with the denv backend selected, and an exported OpenEnv environment passes OpenEnv's reference client checks. Each trainer's own example then runs unchanged, and we publish one such run per trainer, with the denv version recorded, as documentation rather than as tests.
Status
Ships today. Harbor backend and OpenEnv provider, which is what every row of the map reaches. A trainer that already consumes Harbor datasets with a -e setting, or OpenEnv environments with a provider, can train against a denv environment now.
Ships today (2026-10-11). denv verify as the reward path.
Committed by this post. denv export --harbor and --openenv; promote gates; rollout TTLs; one published example run per trainer in the map with its denv version; a denv backend entry in rLLM's and OpenEnv's backend lists.