> ## Documentation Index
> Fetch the complete documentation index at: https://docs.boxd.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# CI runner benchmarks

> The CI of 10 open-source projects on GitHub-hosted runners, Blacksmith, Vercel CI and boxd, every machine 2 vCPU. boxd is 6× faster than one GitHub-hosted job, 5× faster than one Blacksmith job and 7× faster than one Vercel CI task.

Each project ran seven ways: one job and the same shards on GitHub-hosted runners, the same two on [Blacksmith](https://www.blacksmith.sh) runners, the same two on [Vercel CI](https://vercel.com/docs/ci), and the [CI runners setup](https://docs.boxd.sh/use-cases/ci-runners). boxd's speed-up, as the median across the 10 projects:

| boxd against | One job | Sharded |
| - | - | - |
| GitHub-hosted | **6×** | **3×** |
| Blacksmith | **5×** | **2×** |
| Vercel CI | **7×** | **2×** |

## Against GitHub-hosted runners

| Project | boxd | vs one job | vs sharded | GitHub, one job | GitHub, sharded | Tests |
| - | - | - | - | - | - | - |
| [Django](https://github.com/django/django) | **65s** | 12× | 2.9× | 12m 54s | 3m 06s | 19,899 |
| [Prettier](https://github.com/prettier/prettier) | **55s** | 11× | 2.8× | 9m 38s | 2m 36s | 102,413 |
| [GitHub CLI](https://github.com/cli/cli) | **18s** | 7× | 7× | 2m 10s | 2m 02s | 8,445 |
| [ripgrep](https://github.com/BurntSushi/ripgrep) | **11s** | 7× | 12× | 73s | 2m 09s | 1,222 |
| [Excalidraw](https://github.com/excalidraw/excalidraw) | **89s** | 7× | 1.8× | 9m 48s | 2m 39s | 2,456 |
| [React](https://github.com/facebook/react) | **2m 08s** | 6× | 1.7× | 13m 03s | 3m 33s | 6,990 |
| [Vue](https://github.com/vuejs/core) | **25s** | 6× | 3.4× | 2m 32s | 86s | 3,765 |
| [Pydantic](https://github.com/pydantic/pydantic) | **29s** | 5× | 4.1× | 2m 35s | 1m 58s | 13,509 |
| [Zod](https://github.com/colinhacks/zod) | **72s** | 4.1× | 1.8× | 4m 54s | 2m 06s | 8,320 |
| [Svelte](https://github.com/sveltejs/svelte) | **2m 36s** | 2.5× | 1.4× | 6m 24s | 3m 38s | 7,879 |

## Against Blacksmith

Blacksmith runs GitHub Actions jobs on its own machines, with a cache next to the runners. These are the same workflows as on GitHub, with only the runner label changed.

| Project | boxd | vs one job | vs sharded | Blacksmith, one job | Blacksmith, sharded |
| - | - | - | - | - | - |
| Django | **65s** | 8× | 2.0× | 8m 35s | 2m 08s |
| Prettier | **55s** | 8× | 2.3× | 7m 47s | 2m 05s |
| GitHub CLI | **18s** | 20× | 6× | 5m 53s | 1m 49s |
| ripgrep | **11s** | 5× | 6× | 56s | 65s |
| Excalidraw | **89s** | 6× | 1.5× | 8m 20s | 2m 13s |
| React | **2m 08s** | 5× | 1.4× | 11m 05s | 2m 56s |
| Vue | **25s** | 5× | 3.2× | 2m 13s | 80s |
| Pydantic | **29s** | 6× | 3.0× | 2m 47s | 86s |
| Zod | **72s** | 2.6× | 1.9× | 3m 10s | 2m 19s |
| Svelte | **2m 36s** | 1.9× | 1.0× | 5m 02s | 2m 30s |

## Against Vercel CI

Vercel CI runs Turborepo tasks, each on its own machine, after one shared install. Each project's jobs became tasks: one task for the checks and the tests, or one for the checks and one per test shard, on Vercel CI's default of 2 CPUs and 4 GB.

| Project | boxd | vs one task | vs sharded | Vercel CI, one task | Vercel CI, sharded |
| - | - | - | - | - | - |
| Django | **65s** | 10× | 2.5× | 11m 06s | 2m 46s |
| Prettier\* | **55s** | 7× | 2.4× | 6m 22s | 2m 09s |
| GitHub CLI | **18s** | 39× | 15× | 11m 40s | 4m 21s |
| ripgrep | **11s** | 7× | 7× | 73s | 81s |
| Excalidraw | **89s** | 6× | 1.3× | 8m 48s | 1m 54s |
| React | **2m 08s** | over 7× | 2.0× | did not finish | 4m 17s |
| Vue | **25s** | 8× | 4.1× | 3m 13s | 1m 43s |
| Pydantic | **29s** | 6× | 2.5× | 2m 49s | 73s |
| Zod | **72s** | 4.2× | 2.0× | 4m 59s | 2m 26s |
| Svelte | **2m 36s** | 3.4× | 1.8× | 8m 53s | 4m 41s |

React's one task never finished. Vercel CI stops a task after 15 minutes, and all four attempts ran into that limit.

\* Prettier's tasks that run `patterns-dirs.js` have 4 CPUs and 8 GB. That suite starts many CLI processes at once, and with 4 GB the out-of-memory killer ends them.

Each run is the project's lint, format, typecheck and test jobs on a push to `main`. Times are medians, from the push to the result.

## Details

> **Where the speed-up comes from**
>
> * **Sharding** helps the big suites on any runner. Django drops from 12m 54s to 3m 06s on GitHub's own runners. It does nothing for small ones. Every GitHub or Blacksmith job repeats checkout, toolchain setup and install, so ripgrep gets slower when split.
> * **The snapshot** is the rest, the "vs sharded" columns. A boxd job starts with dependencies, build output and caches already on disk, so a shard is running tests a few seconds after the push.
> * **Faster runners** close little of the gap. Blacksmith finishes the same jobs 17 to 19% sooner than GitHub, and each of its jobs still starts from a fresh checkout.
> * **Task caching** is Vercel CI's version of the snapshot, at a coarser grain. A task whose inputs didn't change comes back whole from Turborepo's cache. A task whose inputs changed starts from the installed repository with no tool caches, so Go and Rust compile everything again.
> * **Machine size** matters little. On 8 vCPU machines with 8 shards, 6 of the 10 projects finished within 16% of their 2 vCPU time. Prettier, React and Excalidraw needed 2 to 4 times the shards on 2 vCPU to come close.

> **How this was measured**
>
> * **Projects.** Each at its latest commit on 8 October 2026, copied into a private repository. Excalidraw is at `53973c3`.
> * **Scope.** The project's Linux lint, format, typecheck and unit-test jobs, with one toolchain version. Left out: Windows and macOS jobs, end-to-end suites, jobs that need external services, release and docs jobs.
> * **GitHub, one job.** `ubuntu-latest` in a private repository (2 vCPU, 8 GB), with the ecosystem's standard setup action and dependency cache. One warm-up run to fill the cache, then two measured runs.
> * **GitHub, sharded.** The same runners and caches, as one checks job plus one job per test shard. Two measured runs.
> * **Blacksmith.** The GitHub workflows, one job and sharded, with `runs-on` changed to `blacksmith-2vcpu-ubuntu-2404` (2 vCPU, 8 GB). The standard cache actions use Blacksmith's own cache without changes. Sticky disks and Docker layer caching are left out. One warm-up run, then three measured runs of each.
> * **Vercel CI.** The same commands as Turborepo 2.11.10 tasks tagged `ci`, in copies of the same commits. One task for the checks and the tests, or a checks task plus one task per shard, each on the default 2 CPUs and 4 GB with the Ubuntu image. Bootstrapping installs the toolchains and dependencies into the repository on 4 CPUs. One warm-up run, then three measured runs of each.
> * **boxd.** The [CI runners setup](https://docs.boxd.sh/use-cases/ci-runners) with 2 vCPU / 8 GiB machines: checks on one, each test shard on its own, all created from the snapshot. One warm-up run, then three measured runs.
> * **Shards.** 8 per project. Prettier uses 16. React and Excalidraw use 32 on boxd and 16 on GitHub, where the plan used for the test runs 20 jobs at once. Blacksmith ran them at both counts and finished sooner on 16, so its table uses 16. Vercel CI uses boxd's counts.
> * **Commands.** The same on every platform, apart from the shard flag and on-disk cache flags (`eslint --cache`, `tsc --incremental`) on boxd.
> * **Push to main.** A commit with no code change, the steady state between merges. On Vercel CI the commit changes one file that every task hashes, so every task runs.
> * **Timing.** Wall time from the push to the result, or from the workflow dispatch on GitHub and Blacksmith, median of the measured runs. GitHub and boxd ran on 8 October 2026, Blacksmith and Vercel CI on 28 October 2026.

> **What a run costs**
>
> All at list prices. GitHub charges \$0.006 per runner minute, billed per job and rounded up to the minute. Blacksmith charges \$0.004 per runner minute, counted the same way here, since its pricing doesn't say how it rounds. Vercel CI charges \$0.0035 per CPU-minute on Pro, per second, for every task and for bootstrapping, which runs on 4 CPUs. boxd charges \$0.059 per vCPU-hour and \$0.018 per GiB-hour, metered per second. The boxd figure assumes every machine holds all 8 GiB for the whole run, so it is an upper bound.
>
> | Project | boxd | GitHub, one job | GitHub, sharded | Blacksmith, one job | Blacksmith, sharded | Vercel CI, one task | Vercel CI, sharded |
> | - | - | - | - | - | - | - | - |
> | Django | **4.1¢** | 7.8¢ | 10.5¢ | 3.6¢ | 5.6¢ | 8.4¢ | 6.8¢ |
> | Prettier | **6.5¢** | 6.0¢ | 15.9¢ | 3.2¢ | 7.6¢ | 8.9¢ | 8.9¢ |
> | GitHub CLI | **1.0¢** | 1.8¢ | 10.5¢ | 2.4¢ | 6.0¢ | 8.3¢ | 19.3¢ |
> | ripgrep | **0.6¢** | 1.2¢ | 8.7¢ | 0.4¢ | 3.6¢ | 1.1¢ | 5.5¢ |
> | Excalidraw | **21.0¢** | 6.0¢ | 24.3¢ | 3.6¢ | 11.4¢ | 6.4¢ | 12.6¢ |
> | React | **30.2¢** | 8.4¢ | 24.9¢ | 4.0¢ | 13.4¢ | did not finish | 26.3¢ |
> | Vue | **1.5¢** | 1.8¢ | 6.3¢ | 1.2¢ | 4.0¢ | 2.5¢ | 3.6¢ |
> | Pydantic | **1.8¢** | 1.8¢ | 9.3¢ | 1.2¢ | 4.4¢ | 2.2¢ | 3.5¢ |
> | Zod | **4.5¢** | 3.0¢ | 11.4¢ | 1.2¢ | 7.2¢ | 3.7¢ | 9.3¢ |
> | Svelte | **10.0¢** | 4.2¢ | 8.4¢ | 2.0¢ | 4.8¢ | 6.5¢ | 8.4¢ |
>
> Sharding trades compute for time on every platform. boxd costs less than the sharded GitHub run for 8 of the 10 projects, because no shard pays for setup. Against Blacksmith's sharded run it costs less for 7, or for 5 if Blacksmith bills per second. Against Vercel CI's sharded run it costs less for 7. React and Excalidraw run 33 machines each on boxd, twice the shard count on GitHub and Blacksmith, and the same count as on Vercel CI.

> **Caveats**
>
> * **An unchanged commit is boxd's best case.** The snapshot holds build output and tool caches as well as dependencies, and a real change invalidates part of them. GitHub and Blacksmith carry only dependency caches, so a change costs them less.
> * **Sharding has limits.** One slow test module sets Django's time at any shard count. Svelte gains least: one test file holds most of the suite's time, and its build step has no cache.
> * **Checks run in sequence** on the first machine. For Zod they take as long as the tests, so more shards change nothing.
> * **Go caches test results.** On an unchanged commit, GitHub CLI on boxd mostly replays results cached in the snapshot, and GitHub replays 216 of its 250 test packages from the setup-go cache. Blacksmith runs Go's build cache through its own service, which replayed none, so the single job is slower there than on GitHub. Vercel CI keeps no Go cache between runs, so it compiles and runs all 250 packages with `-race` every time. The row measures cache coverage more than test speed.
> * **Blacksmith runners vary.** Jobs ran on AMD EPYC and, for 3 of 64 runs, Intel Xeon machines. React's single job took between 541s and 955s, and in 5 of the 64 runs one job waited 80 to 101 seconds for a runner.
> * **Retries and skips.** Vue and Zod run with `--retry=2` on every platform. Excalidraw retries only on boxd, and GitHub, Blacksmith and Vercel CI run its tests without retries. A retry only adds time when a test fails, so it never makes a run faster. Django skips one timing-sensitive test and React one test file (4 tests). React's single hosted job needs two extra Jest flags to fit in the runner's memory.
> * **Vercel CI caches whole tasks.** On a commit that changes no file, every task comes back from Turborepo's cache and the run is green in 21 to 27 seconds. Each job here is one task over the whole repository, so any code change runs all of them again, and that is what the Vercel CI runs measure.
> * **Vercel CI stops a task after 15 minutes.** React's one task hit the limit in all four attempts. Excalidraw's finished once with 22 seconds to spare.
> * **A Vercel CI task has 2 GB per CPU.** GitHub's and Blacksmith's 2 vCPU runners have 8 GB. Besides Prettier's 4 CPU tasks, its knip step runs with `KNIP_DISABLE_RAW_TRANSFER=1`, so the parser's buffer fits in 4 GB.
> * **Vercel CI runs tasks as root.** Django's tests run as a separate user there, and GitHub CLI skips `TestMigrationWriteErrors`, which expects a write to fail. Zod declares pnpm as its package manager there, because Turborepo can't start nub, and nub still runs every command.
> * **Vercel CI's machines vary.** Tasks ran on Intel Xeon at 2.5 or 2.9 GHz, which moved Excalidraw's one task between 522 and 733 seconds. Twice, once in a measured run and once in a warm-up, two shards started 180 seconds late. Excalidraw's one task runs without `--retry`, as on GitHub, and 1 of its 4 attempts failed on a timing-sensitive clipboard test.
> * **A snapshot carries state.** Prettier's lint leaves a format cache that its own CLI tests race to delete, so the shards remove it first. Pydantic's plugin tests depend on which files share a shard.

> **Where the time goes on boxd (Excalidraw)**
>
> | Step | Time |
> | - | - |
> | Create 33 machines from the snapshot, all at once | about 10s (4s for 9) |
> | Fetch and check out the commit | about 1s |
> | Install | 0.2s, skipped: the lockfile didn't change |
> | Prettier, ESLint, typecheck, with the caches from the snapshot | 5s, 2s, 5s |
> | 32 test shards, in parallel | 37s at the median, 80s for the slowest |
>
> On the hosted runner the same job spends 50 to 80 seconds on setup (cache restore and install) and runs the tests in 348 to 537 seconds.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.