Reproducible RL environments

Reinforcement learning runs the same environment thousands of times, and every episode should begin from an identical state, so that a difference between two runs comes from the policy rather than from the machine underneath it. Reproducing that state by rebuilding it (reinstall, reload, re-seed every library) is slow and fragile, since one unseeded source of randomness or one leftover file breaks the comparison. boxd takes the reconstruction out of the loop. A fork copies a running machine in milliseconds, memory and disk together, so every rollout starts from literally the same bytes: simulator initialised, weights resident, caches warm. What the seed controls is your policy, and what the fork guarantees is the environment.

Eight isolated rollouts in 1.6 seconds, then two forks proving the same seed gives the same trajectory.

What the primitives give you

Isolation per rollout. Each fork is a hardware-isolated microVM with its own kernel, disk, and network identity. Nothing leaks between workers.
Byte-identical starts. Every fork of the same source continues from the same instruction, with the same processes, memory, and files.
Real-time branching. A fork copies live memory directly, machine to machine, with no snapshot written to disk first. When rollouts branch on the fly, forking skips the save-and-restore round-trip, which keeps your GPUs busy instead of waiting on environment resets.
Stateful resets. A checkpoint rewinds one machine to a saved moment in place, so episode N+1 starts exactly where episode N did.
A durable baseline. A snapshot turns the prepared environment into a named, versioned image that outlives the machine and stays comparable across a whole training run.

Prepare one warm baseline

Set the environment up once, on one machine:

boxd machine new rl-base
boxd machine exec rl-base -- 'pip install gymnasium numpy'
boxd machine exec rl-base -- 'python3 warm_up.py'    # load the sim, weights, and caches

The setup cost is paid exactly once. Every rollout after this point skips it.

Fork per rollout

Each rollout forks the baseline, runs its episode, reports its result, and disappears. The Python and TypeScript SDKs drive this from your training loop:

Python
TypeScript

from concurrent.futures import ThreadPoolExecutor
from boxd import Boxd

boxd = Boxd()   # reads BOXD_API_KEY; authenticates automatically inside a VM

boxd.machines.files.upload(
    "rl-base", "/home/boxd/rollout.py", open("rollout.py").read()
)

def rollout(name: str, seed: int) -> str:
    fork = boxd.machines.fork("rl-base", name)
    boxd.machines.wait_until_ready(fork.id)
    result = boxd.machines.exec(
        fork.id, f"python3 /home/boxd/rollout.py --seed {seed}"
    )
    boxd.machines.delete(fork.id)
    return result.stdout.strip()

with ThreadPoolExecutor(max_workers=8) as pool:
    results = list(pool.map(
        lambda s: rollout(f"rollout-{s}", seed=s), range(8)
    ))

import { readFileSync } from "node:fs";
import { Boxd } from "@boxd-sh/sdk";

const boxd = new Boxd();   // reads BOXD_API_KEY; authenticates automatically inside a VM

await boxd.machines.files.upload(
  "rl-base", "/home/boxd/rollout.py", readFileSync("rollout.py", "utf8")
);

async function rollout(name: string, seed: number): Promise<string> {
  const fork = await boxd.machines.fork("rl-base", { name });
  await boxd.machines.waitUntilReady(fork.id);
  const result = await boxd.machines.exec(fork.id, {
    command: `python3 /home/boxd/rollout.py --seed ${seed}`,
  });
  await boxd.machines.delete(fork.id);
  return result.stdout.trim();
}

const results = await Promise.all(
  [...Array(8).keys()].map((s) => rollout(`rollout-${s}`, s))
);

Forks are copy-on-write, so dozens of them cost close to no extra storage until they start to diverge, and each one lands in about 160ms with the baseline’s full state.

Verify reproducibility

The check is two forks with the same seed:

a = rollout("check-a", seed=42)
b = rollout("check-b", seed=42)
assert a == b, "same starting bytes + same seed should give the same trajectory"

Both forks start from identical machine state, so any remaining difference comes from your own code (an unseeded RNG in the policy, wall-clock reads, network calls) rather than from the environment.

Reset between episodes

To reuse one machine across episodes instead of forking per rollout, save a checkpoint at the starting state and rewind to it:

boxd machine checkpoint save rollout-1 episode-start
# run an episode, let the policy change the environment
boxd machine checkpoint restore rollout-1 episode-start -y

Restore rewinds the machine in place, keeping its name and URL, and brings it back byte-identical to the captured moment. That is what makes the next episode comparable to the last.

A baseline that outlives the machine

A checkpoint belongs to its machine and disappears with it. When a baseline needs to survive, or a teammate needs the same starting point, save a snapshot:

boxd snapshots save rl-base env-v3
boxd machine new rl-base-2 --from-snapshot env-v3

Snapshots are named and versioned, so env-v3 still means the same environment a month later. This is how a result stays reproducible after the original machine is gone.

Scaling out

Rollouts are independent, so they scale sideways. Idle machines suspend and hibernate on their own, which keeps a pool affordable between batches, and a suspended worker wakes in under a millisecond when the next batch starts. Accounts start at 50 concurrent machines, extendable on request for fleet-sized runs. See Resources and limits. When a policy executes untrusted or generated code, create the workers with --isolated so a rollout can reach nothing beyond its own machine. See Sandboxes.

Design notes

Why fork instead of re-seed. Seed-based reproducibility asks every library in the stack to cooperate, and one forgotten RNG quietly breaks it. A fork moves the problem out of your code: the starting state is identical because it is the same memory, so seeds only need to cover the policy’s own choices. Why the baseline stays warm. A fresh machine would reinstall dependencies and reload weights for every rollout, and each of those steps is both wasted time and a chance for two episodes to start from slightly different state. The fork inherits the finished setup, so per-rollout cost is the fork itself. Fork, checkpoint, or snapshot. Fork when you want many parallel copies of one moment. Checkpoint when one machine should return to its own past. Snapshot when the baseline must outlive the machine and stay pinned across a training run.

Getting started

Guides

Use cases

References

Reproducible RL environments

What the primitives give you

Prepare one warm baseline

Fork per rollout

Verify reproducibility

Reset between episodes

A baseline that outlives the machine

Scaling out

Design notes

​What the primitives give you

​Prepare one warm baseline

​Fork per rollout

​Verify reproducibility

​Reset between episodes

​A baseline that outlives the machine

​Scaling out

​Design notes

What the primitives give you

Prepare one warm baseline

Fork per rollout

Verify reproducibility

Reset between episodes

A baseline that outlives the machine

Scaling out

Design notes