Benchmark · Task 1
Sandbox time to first command
From the moment you ask a provider for a brand new sandbox until echo ok comes back with exit code 0. Each provider gets 5 fresh sandboxes on its own default image.
01Results
Cold create to first command
- 1Cloudflare Sandbox483 ms×1.00
- 2Freestyle976 ms×2.02
- 3Tensorlake1.92 s×3.97
- 4BoxLite Cloud4.07 s×8.42
- 5Upstash Box5.02 s×10.39
- 1Cloudflare Sandbox467 ms×1.00
- 2Freestyle897 ms×1.92
- 3Tensorlake1.90 s×4.06
- 4Upstash Box2.76 s×5.90
- 5BoxLite Cloud3.75 s×8.03
- 1Cloudflare Sandbox888 ms×1.00
- 2Freestyle1.57 s×1.77
- 3Tensorlake2.49 s×2.80
- 4BoxLite Cloud4.43 s×4.98
- 5Upstash Box11.67 s×13.13
Measured from one client over the public Internet. 6 more providers could not be measured yet; they are listed below with the reason, never as a zero.
02All metrics
Milliseconds, p50 with min to max underneath. Only successful replicates count toward the summary. Hover a row in the chart above for every sample.
| Provider | OK | cold_start | spawn | first exec after create | teardown |
|---|---|---|---|---|---|
| Cloudflare Sandbox cloudflare/debian-trixie (lite) | 5/5 | 483.0 467.1 to 888.4 | 310.7 297.7 to 716.3 | 172.3 169.4 to 202.5 | 235.0 198.8 to 261.3 |
| Freestyle freestyle default VM image | 5/5 | 975.6 896.6 to 1572.1 | 659.6 590.4 to 1272.0 | 304.0 296.8 to 316.1 | 379.3 362.9 to 455.7 |
| Tensorlake tensorlake default image (tensorlake/ubuntu-minimal) | 5/5 | 1918.8 1895.1 to 2490.0 | 851.4 842.3 to 1372.2 | 1063.0 1052.8 to 1117.8 | 221.6 218.9 to 226.0 |
| BoxLite Cloud boxlite cloud 'base' image | 5/5 | 4067.3 3752.2 to 4428.2 | 2336.8 2093.9 to 2683.7 | 1737.6 1462.4 to 1752.1 | 295.2 283.4 to 560.2 |
| Upstash Box upstash box default runtime (Debian 12 + Node) | 5/5 | 5019.3 2756.1 to 11667.4 | 4742.8 2488.1 to 11395.9 | 276.5 268.0 to 284.2 | 1685.0 499.9 to 2651.6 |
Provider notes
- Cloudflare Sandbox: Timed from this host through the bench Worker in ./cloudflare, so it includes one extra laptop-to-Worker hop.
- Freestyle: Created with an empty firewall (no network), matching Cloudflare's enableInternet: false.
- Tensorlake: REST API (the TS SDK breaks under Bun). POST /sandboxes waits until running by default, so spawn_ms covers boot. Free tier allows 1 concurrent sandbox.
- BoxLite Cloud: BoxLite Cloud on its 'base' image. Cloud create() returns with the box already running; start() is a no-op confirm.
- Upstash Box: SDK Box.create() waits for ready: if POST /v2/box returns 'creating' it polls GET /v2/box/:id every 2s, so spawn_ms may be quantized to ~2s steps.
03Not measured
A provider that could not be measured is shown here with the reason, not ranked as zero or last.
| Provider | Status | Why | Checked |
|---|---|---|---|
| E2B | blocked | New workspace is blocked until a $5.00 card verification payment is made. "This workspace is blocked until it is verified. Make a verification payment to keep using E2B." | 2026-11-11 |
| Runloop | blocked | Signup is invite only. "Registration is currently by invitation only. Please email support@runloop.ai to join the waitlist." | 2026-11-11 |
| boat | blocked | API key works, but creating a sandbox needs a started plan (card checkout). "402 billing_required: Start the $20/month Boat plan to create sandboxes." | 2026-11-12 |
| Novita | blocked | Account has no balance yet. Novita's docs say the free credit unlocks after an onboarding survey. "SandboxError: 402: insufficient balance" | 2026-11-12 |
| Celesto | pending | The API key in use is read-only. A key with the run-commands permission is needed. "403: This API key needs 'Create, start, stop computers and run commands' permission" | 2026-11-12 |
| Daytona | pending | Adapter written, no API key yet. | 2026-11-12 |
04Method
- Start a monotonic clock, then call the provider's create.
- Create a fresh sandbox on the vendor's default image. No custom templates, no warm pools we control.
- Run
echo okuntil it exits 0. Up to 40 tries, 250 ms apart, 2 s per try. - Run
uname -aonce as a second check. Not part of the headline number. - Destroy the sandbox. This always runs, even after a failure.
| Metric | Meaning |
|---|---|
| cold_start_ms | Headline. Clock start until echo ok exits 0. |
| spawn_ms | Clock start until the create call returns a handle. |
| time_to_first_exec_ms | Create returned until echo ok exits 0. |
| teardown_ms | Wall time of the destroy call. |
Only cold_start_ms is fair across providers. Some SDKs wait inside create until the sandbox is up, others return right away, so the split between spawn and first exec says more about the SDK than the platform.
Metric definitions follow StarSling's lifecycle_* metrics inhpc-sandbox-benchmarks (MIT, Copyright (c) 2026 StarSling). This is a small independent harness, not StarSling's, and not an official StarSling result.
05Caveats
- Every number includes the network round trip from one client machine (darwin 27.0.0) to each provider. Your location will shift them.
- Default sizes differ by vendor. This is not StarSling's 4 vCPU / 8 GiB / 40 GB toolchain shape, so it is not comparable to their pipeline tables.
- Cloudflare sandboxes can only be driven from inside a Worker, so that number goes through one small bench Worker and includes that extra hop.
- Upstash's SDK polls for readiness every 2 s inside create, so its numbers move in roughly 2 s steps.
- Five replicates is enough to rank wide gaps, not to separate providers that are within a few hundred ms.
- This measures create to first command from an SDK. It is not signup time, and it says nothing about disk, CPU or pipeline speed.
06Raw data
Every replicate, error and sandbox id behind this page:
- ttfr-20261111-210436-America_Denver.json
- ttfr-20261112-015301-America_Denver.json
- ttfr-20261112-015440-America_Denver.json
- ttfr-20261112-015623-America_Denver.json
Harness: github.com/aiinfraengineer/ttfr-bench. Run it yourself with your own keys.