The previous post said when a lab should put one runtime under its vendors' environments. This one is the same argument with the generality removed: three vendors, what each ships, and what the lab's environment team does with it, with the numbers written down. Vendor names are invented; the shapes are the ones we see.

The numbers marked measured come from denv's conformance runs on a laptop with plain-disk storage. The numbers marked target are what ZFS clones are designed to give and what we will publish once a production ZFS host is running. Everything else is arithmetic.

The three deliveries

Vendor A, "Northwind Ops": incident response on a running shop. A six-service e-commerce stack: Postgres (4.2 GB, 1.8 million orders across 14 tables), Redis with a 40,000-item session store and a job queue, MinIO holding 12 GB of product images and invoices, an API, a worker, and an ops box with the usual tools. 120 tasks: a fault is injected (a stuck migration, a queue consumer that stopped acking, a misconfigured TLS cert, a disk filling), the agent gets a shell and the observability stack, and must diagnose and mitigate. Verifier: a health predicate per task (orders flowing, p95 under a threshold) plus invariants (no table dropped, no alert silenced, no customer data deleted). Delivered as Harbor tasks with a compose file. Price: per task, with a hidden split the lab never sees.

Vendor B, "Ledgerline": month-end close in a data pipeline. A scheduler, a Postgres warehouse (9 GB, 60 models), an object store with 3 GB of raw CSV and parquet drops from four source systems, a transformation layer, and a notebook box. 80 tasks: a source changed its schema, a late-arriving file broke a join, a currency table is stale, duplicates appeared after a retry; the agent must get the close to reconcile. Verifier: deterministic, a reconciliation script with tolerances, plus an invariant that raw files are never edited. Harbor tasks, compose file.

Vendor C, "Deskmate": customer support in a SaaS replica. A ticketing app with Postgres (2 GB, 300,000 tickets and 50,000 customers), a mail stub that records outbound email, a knowledge base of 400 policy documents, and a browser-capable agent box. 200 tasks: resolve a case across the ticket, the customer record and the policy, with escalation rules. Verifier: a rubric judged by a model, plus hard policy checks (no refund over the limit without approval, no PII in outbound mail). Harbor tasks, compose file, the judge as a separate image.

Three vendors, three stacks, 400 tasks, about 30 GB of state between them. Delivered the same week.

Monday: intake

The team's intake is the same for all three because the runtime is the same.

denv import harbor ./deliveries/northwind   # 120 tasks → project northwind, env alpha
denv import harbor ./deliveries/ledgerline  # 80 tasks  → project ledgerline
denv import harbor ./deliveries/deskmate    # 200 tasks → project deskmate

Each import deploys the vendor's compose file as alpha, runs the vendor's warm step (Northwind replays a day of orders so the queue has a backlog; Ledgerline loads the four drops; Deskmate seeds the tickets), and takes the snapshot. The deploy record for Northwind looks like this, trimmed:

resolve     ok   digests pinned: 6 images
prepare     ok   datasets: pg, redis, minio, home
start       ok   started_in_ms: 41200
health      ok   6/6
warm        ok   "replay-orders.sh --day 2026-09-30"
snapshot    ok   datasets: 4, frozen_for: 3.1 s, consistent: true,
                 digests: {pg: sha256:7930…, redis: sha256:4c1a…, minio: sha256:0e77…, home: sha256:b2d9…}
verdict     ok   version ver_A

Intake also runs the five checks automatically, because resolve refuses an agent service that is root and the verdict records the egress probe. Deskmate's first delivery fails at resolve: the agent box has no user. The vendor adds user: "1000:1000" and redelivers in the afternoon. That exchange used to take a week of email.

Time on Monday: about four hours of machine time across the three worlds, almost all of it in the warm steps. Human time: an hour.

Tuesday: the hidden split

The lab has its own held-out tasks per vendor: 30 for Northwind, 20 for Ledgerline, 50 for Deskmate, written against the vendors' worlds but never shared. Each task runs with five repeats to get a noise bound, with four agent harnesses.

Vendor	Tasks × repeats × harnesses	Forks
Northwind	30 × 5 × 4	600
Ledgerline	20 × 5 × 4	400
Deskmate	50 × 5 × 4	1,000

Two thousand forks of three versions. On plain disk, Northwind's 16 GB fork takes minutes each; that is why this used to be done on a subset. On ZFS a fork is a clone and the target is about a second regardless of size, so the whole split runs in one evening on two bare-metal boxes.

Every run is a Harbor trial with denv as the backend:

harbor run --dataset lab/northwind-hidden@v1 --agent claude-code -k 5 -e denv.harbor:DenvEnvironment

Each trial's record carries the version, the fork's digests, the probe result and the verifier's output. The eval ledger is a query over records, not a spreadsheet of what someone remembers running. Two forks of ver_A that disagree can be compared by digest first; if the digests match, the difference is the agent.

Result on Tuesday night, for the decision the team actually has to make:

Vendor	Best harness pass@1	Noise bound (5 repeats)	Verdict
Northwind	38 %	± 2.9 pp	in range; buy
Ledgerline	71 %	± 1.4 pp	too easy at this price; renegotiate difficulty mix
Deskmate	44 %	± 6.2 pp	noise too high; the judge is unstable, send back

Deskmate's noise is the judge, not the world: the forks' digests are identical and the harnesses' trajectories are not. The team sends the vendor the twenty records where the same trajectory got two different scores. That is a conversation with evidence instead of a feeling.

Wednesday: the attacks

The team's reward-hacking probes are a set of scripted attackers plus two hacker agents, run on forks of each version with the vendor's own verifier.

File-level attacks never reach a grader on denv: the submission is copied out of the frozen agent box with symlinks refused, the grader runs on a separate fork as its own principal, and the record says what was refused. On the three worlds the scripted set produced 47 refusals and zero successful file-level hacks, in about twenty minutes of forks. The team did not write those probes this time; they are the same ones every time, and the runtime answers them.

The semantic attacks are where the week goes. Northwind: a hacker agent found that silencing the alert and restarting the consumer passed the health predicate; the invariant check caught the silence, so the task held. Ledgerline: a hacker agent passed one reconciliation by editing a raw file; the "raw files never edited" invariant caught it on the fork's digest comparison, and the task held. Deskmate: a hacker agent issued a refund over the limit by splitting it into three; the policy check did not sum refunds per ticket. That is a real hole, found with a record attached, and it goes back to the vendor as a task-level defect with the trajectory.

Forks on Wednesday: about 900. Human time: a day, all of it on the semantic findings.

Thursday: training

Northwind is accepted. The lab's trainer reaches the environment through the Harbor taskset it already uses:

env = vf.load_environment("harbor", dataset="northwind/incidents@v1", agent="claude-code",
                          environment="denv.harbor:DenvEnvironment")

Eight samples per prompt, 120 prompts, 200 steps: 192,000 rollouts over the run. Each rollout forks ver_A, runs the agent with caps and egress denied, grades on a fork as a separate principal, and reports a reward with a record. On rebuilt-per-episode sandboxes the 16 GB world would make this run infeasible; on ZFS clones the environment side is two bare-metal boxes.

	Rebuild per episode	Fork per episode (ZFS, target)
Environment time per rollout	40 to 90 s	≈ 1 s
Environment hours for the run	2,100 to 4,800	≈ 55
What the GPUs do meanwhile	wait	train

Friday: a vendor update

Northwind ships ver_B: a new fault family and a fix to a flaky healthcheck. The lab does not point training at it until it passes gates:

denv promote northwind alpha staging
  gate.fork          ok   fresh fork of ver_B
  gate.consistency   ok   4 datasets agree (txg 9918)
  gate.replay        ok   5/5 identical, digests equal
  gate.invariants    ok   0 deleted, 0 disabled, 0 silenced
  gate.soak          ok   held through the next scheduled trigger
  apply              ok   staging → ver_B
  verdict            ok   sheet written

The gate record is the data sheet the vendor would otherwise write by hand. If gate.replay had failed, the team would know that a score change next week was the world, not the model, before spending a GPU-hour on it.

The week, added up

	Before	With denv
Runtimes to understand	3	1
Diligence per delivery	a week of checks per vendor	the five checks are in every record
Hidden split	a subset, on the vendor's provider	2,000 forks in an evening
File-level reward hacks found by the team	a few, by hand	0 reach a grader; 47 refused automatically
Semantic hacks found	some	3, each with a trajectory and a record
Vendor conversations	opinions	records
Training run environment cost	thousands of sandbox-hours	two machines

Human time across the week: three days, nearly all on the content of the environments. Machine time: two bare-metal boxes with ZFS, in the lab's account.

Status

Everything in Monday through Wednesday that touches the runtime (import, deploy, warm, snapshot with digests, fork, verify with refusals, the Harbor backend) is shipped and covered by the conformance suite, with the exception of denv import harbor, which today is a compose file plus denv deploy. The ZFS fork time is a target until the production host is measured. Friday's promote gates are specified in the blog and not yet built. The vendors and their numbers are a scenario; the shapes of the worlds are the ones we see.