Grade an episode with a separate verifier

1. Declare the agent service and the verifier

x-denv:
  kind: rl-environment
  agent:
    service: box                       # must set a non-root user
  verifier:
    image: ghcr.io/you/grader:1.2      # or `service: <name>` to grade inside a service of the fork
    command: ["/grade.sh"]
    collect: ["/home/agent/answer.txt", "/home/agent/out"]
    timeout_sec: 300

collect paths are absolute paths inside the agent service. They are copied to a root-owned directory before anything is graded.

2. Write the grader

The grader sees the copy at /submission, read-only, and writes the reward:

#!/bin/sh
# /grade.sh inside ghcr.io/you/grader:1.2
mkdir -p /logs/verifier
if diff -q /submission/home/agent/answer.txt /expected.txt >/dev/null; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi

If reward.txt is absent, the last line of stdout is parsed as the reward. The grader runs on the fork's network, so it can query the fork's database too; it runs as root in its own container, with the agent's processes already stopped.

3. Run it

denv verify <env>
denv verify --resume <env>    # unfreeze the agent afterwards

From Python:

from denv import Client
v = Client().verify(env_id)
print(v.reward, v.source, v.refused)

What the record shows

Field	Meaning
reward	the number the grader produced; null if the verify failed
source	reward.txt, stdout, or refused
refused	paths that were not regular files or directories, with the reason: symlink, fifo, device, socket, or over the size and depth limits. Any refusal scores 0 and the grader does not run
missing	declared paths that did not exist
agent_frozen	whether the agent's container was left frozen for inspection

The grader's fork is destroyed at the verdict. The agent's environment is left as it was.