Skip to main content
The Inspect wordmark and the AI Security Institute logo inside the boxd card. Inspect AI is the open-source framework for large language model evaluations from the UK AI Security Institute. An eval runs its samples in sandboxes, where the model’s tools run commands and read and write files. boxd-inspect adds the sandbox type boxd. Each sample runs on its own boxd machine, a full Ubuntu 24.04 VM, and the machine is deleted when the sample ends. Tasks with a compose.yaml or Dockerfile run unchanged: Docker runs them inside the machine. Inspect added it to the Sandboxes section of its extensions list (pull request).

Set up

1

Install the package

It needs Python 3.11 or later. Installing it registers the sandbox type boxd with Inspect, so there is nothing to import in your task.
2

Create an API key

Machines are created and billed in the key’s organization. The key is read from the environment only, so it never ends up in the eval log.
3

Run a task on boxd

kernel.py
Any model Inspect supports works.

Run a Docker task on boxd

A task written for Inspect’s Docker sandbox runs on boxd when you override the sandbox type. Its own Dockerfile or compose.yaml is used:
You can also name the file in the task: sandbox=("boxd", "compose.yaml"). The compose file’s directory goes to the machine, so build: contexts next to it work. Several services share one machine, and the service named default (or marked x-default: true) is the default sandbox, as in Docker. Without a compose file, commands run on the machine itself as the boxd user, with passwordless sudo, Docker, Python, Node, Go and git, in /home/boxd.

Configuration

Pass a BoxdSandboxEnvironmentConfig, or a JSON file with the same fields as sandbox=("boxd", "boxd.json"):
Limit how many machines run at once with Inspect’s --max-sandboxes. Your organization’s machine quota is the upper bound, and boxd manage billing shows it.

Clean up

A sample’s machine is deleted when the sample ends, also when it fails or is cancelled. Any machine left over is deleted by id when the eval ends. With --no-sandbox-cleanup the machines stay, and Inspect prints their ids:
Delete one when you are done:
If the eval process is killed, its machines keep running. Set delete_after so boxd deletes them for you. inspect sandbox cleanup boxd without an id deletes every machine labelled inspect-sandbox=boxd in the organization that is not running, and keeps running ones because they can belong to someone else’s eval.

Limits

  • One machine per sample, shared by all services of its compose file.
  • Fresh machines: compose images are pulled for every sample. For big images, prepare a machine once, save a snapshot and pass from_snapshot.
  • Compose directory: up to 512 MiB, sent to the machine as one archive.
  • Command output: 10 MiB per stream, and read_file up to 100 MiB, the same limits as every Inspect sandbox.

Read more