Skip to main content
The banner shows the boxd.sh and Harbor logos, with Harbor described as the agent evaluation framework. Harbor is a framework for evaluating and optimizing AI agents and models. It runs agents like Claude Code, Codex, or OpenHands against benchmarks like Terminal-Bench and SWE-Bench across many parallel sandboxes. boxd is one of Harbor’s built-in environment providers, and it gives every trial its own isolated KVM microVM with a full Docker daemon, so Dockerfile and multi-container docker-compose.yaml tasks run unmodified:
The provider merged upstream after Harbor 0.25.0, so the command above installs Harbor from its main branch. Once a release includes it, uv tool install "harbor[boxd]" installs the same provider from PyPI. Harbor running inside a boxd machine needs no BOXD_API_KEY, because the SDK authenticates from the machine itself. Every machine Harbor creates is isolated. The agent under test keeps outbound internet for model APIs and package registries. It cannot reach your other machines, and it gets none of your connected integrations or agent logins. Isolation is fixed at creation. Tasks that ask for no-network run with the internet cut off as well. Each concurrent trial is one machine, so your organization’s machine quota caps --n-concurrent. boxd manage billing shows the quota.

Start trials from a snapshot

For benchmarks, start from a snapshot. Prepare a machine, pull or build the task image on it, save a snapshot, and create every sandbox from it with --ek from_snapshot=<snapshot>. Each trial then restores a machine whose Docker cache already holds the task image instead of building it from scratch. A snapshot lives in one region, so save it in the region the trials run in. The machine size comes from the snapshot, which means from_snapshot cannot be combined with vcpu or memory.

Provider options

Pass these with --ek <name>=<value>, or under environment.kwargs in a job config. Harbor’s example config documents each one and runs with harbor run -c boxd.yaml. Harbor’s sandbox reference lists what each provider supports, from Docker Compose to network policies and CPU and memory limits.