BenchmarksMedian of 3 runs · nothing staged

48 agents.One game. Three ways to ship it.

The same task, the same 48 agents, the same model. Only the way their work reaches main changes. Every figure is the median, with the fastest and slowest run beside it.

To 100% of tests
17.0 min
vs 37.8 min on GitHub + Actions
Model spend
$4.55
vs $19.94
Pull requests opened
0
saves land live

Same start. Watch the clock.

AgentflareEvery test passing
GitHub + Actions + merge queueEvery test passing
GitHub + Dagger Cloud + merge queueEvery test passing

Solid: building to 80% of tests. Faded: the last 20%. The number marks the median minute every test passed.

Results

Bars show each setup’s median against the slowest median. Shorter is better on every row.

To 80% of tests

GitHub + Actions + merge queue19.3 min18.3 min to 23.4 min
GitHub + Dagger Cloud + merge queue24.4 min22.4 min to 37.4 min

To 100%

GitHub + Actions + merge queue37.8 min28.5 min to 43.5 min
GitHub + Dagger Cloud + merge queue37.5 min34.6 min to never · 2 of 3 got there

Model spend

Agentflare$4.55$4.39 to $9.04
GitHub + Actions + merge queue$19.94$18.25 to $20.35
GitHub + Dagger Cloud + merge queue$17.01$13.04 to $20.34

Agent-minutes blocked

Agentflare51.34.4 to 91.6
GitHub + Actions + merge queue522429 to 960
GitHub + Dagger Cloud + merge queue558398 to 615

Lines written that didn't land

Agentflare37%24% to 54%
GitHub + Actions + merge queue74%68% to 77%
GitHub + Dagger Cloud + merge queue71%68% to 80%

Pull requests opened

AgentflareNonenone: saves land live
GitHub + Actions + merge queue106101 to 110
GitHub + Dagger Cloud + merge queue8078 to 98

How we measured

  • The task. Build Quorum Run, a small browser game, from its design doc. The score is how many of its 333 acceptance tests pass on main, sampled about once a minute.
  • The same everything else. 48 agents in the pi 1.4.0 harness, the same model (GLM 5.3 Flash, glm-5.3-flash:fast), the same prompt, a 90-minute cap. Each run used one of two endpoints for the model, inco or cpa, at the same prices; the table names which. Agents ran on four hosts: two Shadeform CPU VMs (hub 12 agents, vm2 20), a home machine (sparks, 8), and a Mac (8). Agentflare's hosted run h1 put all 48 on three Shadeform CPU VMs instead, 16 each, with the runners alone on the hub.
  • Agentflare. The agents edit together and each save lands on main once no one else holds its lines. A ci action runs the tests on two runners and moves when both agree. Agentflare's score counts only samples where main parses.
  • GitHub + Actions + merge queue. Each agent works in its own clone, pushes a branch, and opens a pull request. GitHub Actions runs the tests, and a merge queue (up to 5 builds at once, squash merge, eject on failure) merges.
  • GitHub + Dagger Cloud + merge queue. The same, with the checks run on Dagger Cloud.
  • Model spend uses GLM 5.3 Flash's prices on both endpoints: $0.30 per million input tokens, $0.06 cached, $1.00 output. Blocked is agent-minutes spent waiting on CI or the queue plus resolving conflicts, not counting waits at the end; for Agentflare it counts waiting commands and waits for the runners, a different instrument, so read it as order of magnitude. Lines that didn't land are the lines agents wrote into the game's code that aren't on the final main.
  • The replay plays each run's own log on one clock that starts at its first agent message. Saves are the agents' file writes. Asks and holds are the agentflare commands they ran, and refusals are the saves their daemons refused. Pull requests move by GitHub's mergeable state, sampled about every 30 seconds, and the queue's log.
  • Three runs each. Each setup's headline is the median of its three runs, with the fastest and slowest beside it. Agentflare's runs are 3, 4, and h1, on the build tested here. In Agentflare runs 3 and 4, 8 of the 48 agents and the score sampler ran on a heavily loaded Mac; they count, with that noted. Where a run never reached a mark, it counts as slowest. The replay plays each setup's median run by time to 80%.
  • Every run is in the table below, and every figure links to its run's timeline. The manifest of runs is here.
SetupRunStartedEndpointAgents ran onTo 80%To 100%SpendNote
Agentflare32026-10-31 13:58 UTCincohub 12 · vm2 20 · sparks 8 · Mac 87.0 min11.2 min$4.398 of the 48 agents and the score sampler ran on a loaded Mac (load 30 to 48 and above on 16 cores).
Agentflare42026-10-31 15:03 UTCincohub 12 · vm2 20 · sparks 8 · Mac 87.5 min17.5 min$9.04replayed8 of the 48 agents and the score sampler ran on a loaded Mac (load 30 to 48 and above on 16 cores). 80% to 100% took 10 minutes: a burst of whole-file rewrites of game.js.
Agentflareh12026-10-31 16:47 UTCinco3 hosted CPU VMs, 16 agents each11.1 min17.0 min$4.55Hosted: every agent on a Shadeform CPU VM (2 vCPU, 8 GB each), the runners alone on the hub.
GitHub + Actions + merge queue12026-10-31 03:24 UTCcpahub 12 · vm2 20 · sparks 8 · Mac 818.3 min43.5 min$19.94
GitHub + Actions + merge queue22026-10-31 06:43 UTCincohub 12 · vm2 20 · sparks 8 · Mac 823.4 min37.8 min$20.35
GitHub + Actions + merge queue32026-10-31 12:24 UTCincohub 12 · vm2 20 · sparks 8 · Mac 819.3 min28.5 min$18.25replayed
GitHub + Dagger Cloud + merge queue12026-10-31 01:42 UTCcpahub 12 · vm2 20 · sparks 8 · Mac 837.4 minnever · 90%$17.01Stopped at 46 minutes, at 90%.
GitHub + Dagger Cloud + merge queue22026-10-31 05:06 UTCcpahub 12 · vm2 20 · sparks 8 · Mac 824.4 min34.6 min$13.04replayedThe endpoint's credit ran out at 37 minutes, after 100%.
GitHub + Dagger Cloud + merge queue32026-10-31 10:48 UTCincohub 12 · vm2 20 · sparks 8 · Mac 822.4 min37.5 min$20.34

Other Agentflare runs

Earlier Agentflare runs in the pi harness, outside the comparison above: each differs from it in model, agent count, or setup, and none sampled the score on main while it ran, so the one measure here is the tests passing on main when it ended. They are not in the medians.

StartedAgentsTopologyHarnessModelTime boundPassing at the endNote
2026-10-30 14:24 UTC5one hostpi 1.4.0qwen3.8-flash-next35 min, 80 turns per agent333 of 3333 agents hit the wall clock and 2 the turn cap; main passed every test at the end.
2026-10-30 15:14 UTC48hub + 4 hostspi 1.4.0qwen3.8-flash-next150 min, 200 turns per agent124 of 33328 agents stopped with their checks passed; 20 hit the wall clock.
2026-10-30 19:36 UTC48hub + 4 hostspi 1.4.0qwen3.8-flash-next (intended)not recordednoneNever started: every agent failed at launch on a misconfigured model provider.
2026-10-30 19:49 UTC48hub + 4 hostspi 1.4.0qwen3.8-flash-nextnot recordednoneStopped after about 82 minutes, before its report and acceptance were written.
2026-10-31 02:38 UTC48hub + 4 hostspi 1.4.0glm-5.3-flash:fast90 min333 of 333All 48 agents stopped with their checks passed.
2026-10-31 06:17 UTC4hub + 2 hostspi 1.4.0glm-5.3-flash:fast10 min0 of 333The agents' logs show no work; nothing passed on main.
2026-10-31 06:24 UTC4hub + 2 hostspi 1.4.0glm-5.3-flash:fast12 min333 of 333All 4 agents stopped with their checks passed.
2026-10-31 08:58 UTC48hub + 4 hostspi 1.4.0glm-5.3-flash:fast90 min333 of 33345 agents stopped with their checks passed; 2 stopped when the Stop gate failed open; 1 hit the wall clock.
2026-10-31 14:06 UTC48hub + 4 hostspi 1.4.0glm-5.3-flash:fast90 min333 of 333All 48 agents stopped with their checks passed.
2026-10-31 15:05 UTC48hub + 4 hostspi 1.4.0glm-5.3-flash:fast90 min333 of 33347 agents stopped with their checks passed; 1 stopped when the Stop gate failed open.
2026-10-31 16:56 UTC48hub + 3 hostspi 1.4.0glm-5.3-flash:fast90 min333 of 333All 48 agents stopped with their checks passed.
2026-10-31 17:44 UTC48hub + 3 hostspi 1.4.0glm-5.3-flash:fast90 min0 of 333The agents' logs show no work, and main held 41 documents where complete runs held 64; nothing passed.