Inspect on denv

Inspect, from the UK AI Security Institute, is the evaluation framework with the cleanest sandbox abstraction: a SandboxEnvironment with exec, read_file, write_file, per-sample setup and cleanup, selected in a task with sandbox="docker" or any plugin name. Plugins register through a Python entry point, so a pip install is enough to add one.

denv is such a plugin.

pip install "denv[inspect]"
export DENV_SERVER=https://api.denv.ai DENV_API_KEY=dak_…

Selecting denv in a task

Inspect treats a string sandbox config as a file path, so configuration lives in a file or a model, not a bare image name.

from inspect_ai import Task, task
from inspect_ai.dataset import json_dataset
from inspect_ai.scorer import includes
from inspect_ai.solver import generate, use_tools
from inspect_ai.tool import bash

@task
def ops_triage():
    return Task(
        dataset=json_dataset("tasks.jsonl"),
        solver=[use_tools(bash()), generate()],
        scorer=includes(),
        sandbox=("denv", "denv.yaml"),
    )

denv.yaml describes the sandbox. Two shapes:

# a single image, like Inspect's docker sandbox
image: python:3.12-slim
egress: deny          # default
volumes: [/work]      # the only paths that persist across snapshot and fork

# a fork of a whole denv environment: services, data, network
from_version: ver_01M4KQ9F1NT5PRPYS4G85X2VCB
service: opsbox       # where exec runs

With sandbox="denv" and no file, the image comes from DENV_SANDBOX_IMAGE (default python:3.12-slim). A denv.yaml next to the task file is picked up automatically, as Inspect does for compose.yaml. Inline config is DenvSandboxConfig(image=…) or DenvSandboxConfig(from_version=…) from denv.inspect.

Per-sample sandboxes work the same way: a Sample may carry sandbox=("denv", "ver_…") to fork a different version for that sample, which is how a dataset mixes worlds.

What happens during an eval

task_init. Credentials are checked so a missing key fails before any model call.

sample_init. One POST /v1/sandboxes per sample. From an image: an environment with one service and volumes for the listed paths. From a version: a fork, data present, about a second on ZFS. The sandbox returns when healthy; the sample starts.

exec. sandbox().exec(cmd, cwd, env, user, timeout, input) becomes one POST /v1/envs/{id}/exec. The argv is run through bash when the image has it, sh otherwise, with env exported and cwd applied. input is written to a file in the sandbox and redirected in. A non-root user is honoured with su. Timeouts are enforced server-side and surface as Inspect's TimeoutError.

write_file / read_file. Tar through PUT and GET /v1/envs/{id}/files. Text and bytes both work.

sample_cleanup. DELETE /v1/envs/{id}, unless --no-sandbox-cleanup, in which case the environment is kept and inspect sandbox cleanup denv reaps it later. Interrupted evals clean up the same way.

connection. inspect eval --interactive and the sandbox().connection() call return denv exec --service <svc> <env> -- sh, so you can attach to a sample's environment from your terminal.

What Inspect gains on denv

• Whole-system samples. An Inspect sandbox is normally one container, or a compose stack rebuilt per sample. With from_version every sample forks a running world with data in its databases. Eval of an incident-response task starts in the incident, not at boot.
• Mid-sample branching. A solver can call denv.Client().snapshot(env_id) and fork from it to evaluate several continuations of the same state. Inspect has no primitive for this; denv exposes it through the client and the sample's sandbox() carries the environment id.
• Protections without Dockerfile work. Caps, egress denial with a recorded probe, and a non-root agent user come from the environment. egress: allow in denv.yaml is the only way a sample reaches the internet, and the eval log shows that it did.
• An eval log that names the environment. Each sample's events include the denv environment id and the version it was forked from. Two evals that disagree can be compared on exactly what world they ran in.

Concurrency

default_concurrency() is 8 sandboxes per eval; raise it with --max-sandboxes. Fork cost on ZFS is roughly constant, so a hundred concurrent samples is a hundred clones, not a hundred builds. On hosted sandbox targets the provider's limits apply and are shown in the console before you start.

Scoring with a separate verifier

Inspect scorers run in the eval process and call back into the sandbox. That is fine for includes() or a model-graded scorer. For a grader that must not share a process tree with the agent, give the task an x-denv.verifier in the environment's compose file and call denv.Client().verify(env_id) from a scorer. denv stops the agent's processes in a fork, collects the declared paths to a root-owned location, refuses special files, and runs the verifier image there. The reward comes back as the score and the verifier's evidence is in the environment's deployment record.

Conformance

make e2e-frameworks runs an Inspect eval with the mockllm/model against a denv sandbox: a solver writes a marker file, a scorer reads it back with exec, accuracy must be 1.0, and the sample's environment must be destroyed afterwards. The plugin is loaded through the inspect_ai entry point only; the test does not import denv.inspect directly, so a packaging mistake fails the test.

Status

Ships today. sandbox="denv", ("denv", "denv.yaml"), ("denv", "ver_…"), DenvSandboxConfig; sandboxes from an image or a version; exec with cwd, env, user, stdin and timeout; read and write files; cleanup; connection string; entry-point registration; conformance with mockllm.

Ships today (2026-10-11). Client().verify() returning the reward and the evidence, for use from a scorer.

Committed by this post. inspect sandbox cleanup denv; environment id and version in the eval log's sample events; --max-sandboxes mapped to provider limits with the console's capability row; GPU samples once device passthrough ships.