48 agents.One game. Three ways to ship it.
The same task, the same 48 agents, the same model. Only the way their work reaches main changes. Every figure is the median, with the fastest and slowest run beside it.
Same start. Watch the clock.
AgentflareEvery test passing
GitHub + Actions + merge queueEvery test passing
GitHub + Dagger Cloud + merge queueEvery test passing
Solid: building to 80% of tests. Faded: the last 20%. The number marks the median minute every test passed.
0.0 min
Results
Bars show each setup’s median against the slowest median. Shorter is better on every row.
To 80% of tests
To 100%
Model spend
Agent-minutes blocked
Lines written that didn't land
Pull requests opened
How we measured
- The task. Build Quorum Run, a small browser game, from its design doc. The score is how many of its 333 acceptance tests pass on main, sampled about once a minute.
- The same everything else. 48 agents in the pi 1.4.0 harness, the same model (GLM 5.3 Flash, glm-5.3-flash:fast), the same prompt, a 90-minute cap. Each run used one of two endpoints for the model, inco or cpa, at the same prices; the table names which. Agents ran on four hosts: two Shadeform CPU VMs (hub 12 agents, vm2 20), a home machine (sparks, 8), and a Mac (8). Agentflare's hosted run h1 put all 48 on three Shadeform CPU VMs instead, 16 each, with the runners alone on the hub.
- Agentflare. The agents edit together and each save lands on main once no one else holds its lines. A
ciaction runs the tests on two runners and moves when both agree. Agentflare's score counts only samples where main parses. - GitHub + Actions + merge queue. Each agent works in its own clone, pushes a branch, and opens a pull request. GitHub Actions runs the tests, and a merge queue (up to 5 builds at once, squash merge, eject on failure) merges.
- GitHub + Dagger Cloud + merge queue. The same, with the checks run on Dagger Cloud.
- Model spend uses GLM 5.3 Flash's prices on both endpoints: $0.30 per million input tokens, $0.06 cached, $1.00 output. Blocked is agent-minutes spent waiting on CI or the queue plus resolving conflicts, not counting waits at the end; for Agentflare it counts waiting commands and waits for the runners, a different instrument, so read it as order of magnitude. Lines that didn't land are the lines agents wrote into the game's code that aren't on the final main.
- The replay plays each run's own log on one clock that starts at its first agent message. Saves are the agents' file writes. Asks and holds are the
agentflarecommands they ran, and refusals are the saves their daemons refused. Pull requests move by GitHub's mergeable state, sampled about every 30 seconds, and the queue's log. - Three runs each. Each setup's headline is the median of its three runs, with the fastest and slowest beside it. Agentflare's runs are 3, 4, and h1, on the build tested here. In Agentflare runs 3 and 4, 8 of the 48 agents and the score sampler ran on a heavily loaded Mac; they count, with that noted. Where a run never reached a mark, it counts as slowest. The replay plays each setup's median run by time to 80%.
- Every run is in the table below, and every figure links to its run's timeline. The manifest of runs is here.
| Setup | Run | Started | Endpoint | Agents ran on | To 80% | To 100% | Spend | Note | |
|---|---|---|---|---|---|---|---|---|---|
| Agentflare | 3 | 2026-10-31 13:58 UTC | inco | hub 12 · vm2 20 · sparks 8 · Mac 8 | 7.0 min | 11.2 min | $4.39 | 8 of the 48 agents and the score sampler ran on a loaded Mac (load 30 to 48 and above on 16 cores). | |
| Agentflare | 4 | 2026-10-31 15:03 UTC | inco | hub 12 · vm2 20 · sparks 8 · Mac 8 | 7.5 min | 17.5 min | $9.04 | replayed | 8 of the 48 agents and the score sampler ran on a loaded Mac (load 30 to 48 and above on 16 cores). 80% to 100% took 10 minutes: a burst of whole-file rewrites of game.js. |
| Agentflare | h1 | 2026-10-31 16:47 UTC | inco | 3 hosted CPU VMs, 16 agents each | 11.1 min | 17.0 min | $4.55 | Hosted: every agent on a Shadeform CPU VM (2 vCPU, 8 GB each), the runners alone on the hub. | |
| GitHub + Actions + merge queue | 1 | 2026-10-31 03:24 UTC | cpa | hub 12 · vm2 20 · sparks 8 · Mac 8 | 18.3 min | 43.5 min | $19.94 | ||
| GitHub + Actions + merge queue | 2 | 2026-10-31 06:43 UTC | inco | hub 12 · vm2 20 · sparks 8 · Mac 8 | 23.4 min | 37.8 min | $20.35 | ||
| GitHub + Actions + merge queue | 3 | 2026-10-31 12:24 UTC | inco | hub 12 · vm2 20 · sparks 8 · Mac 8 | 19.3 min | 28.5 min | $18.25 | replayed | |
| GitHub + Dagger Cloud + merge queue | 1 | 2026-10-31 01:42 UTC | cpa | hub 12 · vm2 20 · sparks 8 · Mac 8 | 37.4 min | never · 90% | $17.01 | Stopped at 46 minutes, at 90%. | |
| GitHub + Dagger Cloud + merge queue | 2 | 2026-10-31 05:06 UTC | cpa | hub 12 · vm2 20 · sparks 8 · Mac 8 | 24.4 min | 34.6 min | $13.04 | replayed | The endpoint's credit ran out at 37 minutes, after 100%. |
| GitHub + Dagger Cloud + merge queue | 3 | 2026-10-31 10:48 UTC | inco | hub 12 · vm2 20 · sparks 8 · Mac 8 | 22.4 min | 37.5 min | $20.34 |
Other Agentflare runs
Earlier Agentflare runs in the pi harness, outside the comparison above: each differs from it in model, agent count, or setup, and none sampled the score on main while it ran, so the one measure here is the tests passing on main when it ended. They are not in the medians.
| Started | Agents | Topology | Harness | Model | Time bound | Passing at the end | Note |
|---|---|---|---|---|---|---|---|
| 2026-10-30 14:24 UTC | 5 | one host | pi 1.4.0 | qwen3.8-flash-next | 35 min, 80 turns per agent | 333 of 333 | 3 agents hit the wall clock and 2 the turn cap; main passed every test at the end. |
| 2026-10-30 15:14 UTC | 48 | hub + 4 hosts | pi 1.4.0 | qwen3.8-flash-next | 150 min, 200 turns per agent | 124 of 333 | 28 agents stopped with their checks passed; 20 hit the wall clock. |
| 2026-10-30 19:36 UTC | 48 | hub + 4 hosts | pi 1.4.0 | qwen3.8-flash-next (intended) | not recorded | none | Never started: every agent failed at launch on a misconfigured model provider. |
| 2026-10-30 19:49 UTC | 48 | hub + 4 hosts | pi 1.4.0 | qwen3.8-flash-next | not recorded | none | Stopped after about 82 minutes, before its report and acceptance were written. |
| 2026-10-31 02:38 UTC | 48 | hub + 4 hosts | pi 1.4.0 | glm-5.3-flash:fast | 90 min | 333 of 333 | All 48 agents stopped with their checks passed. |
| 2026-10-31 06:17 UTC | 4 | hub + 2 hosts | pi 1.4.0 | glm-5.3-flash:fast | 10 min | 0 of 333 | The agents' logs show no work; nothing passed on main. |
| 2026-10-31 06:24 UTC | 4 | hub + 2 hosts | pi 1.4.0 | glm-5.3-flash:fast | 12 min | 333 of 333 | All 4 agents stopped with their checks passed. |
| 2026-10-31 08:58 UTC | 48 | hub + 4 hosts | pi 1.4.0 | glm-5.3-flash:fast | 90 min | 333 of 333 | 45 agents stopped with their checks passed; 2 stopped when the Stop gate failed open; 1 hit the wall clock. |
| 2026-10-31 14:06 UTC | 48 | hub + 4 hosts | pi 1.4.0 | glm-5.3-flash:fast | 90 min | 333 of 333 | All 48 agents stopped with their checks passed. |
| 2026-10-31 15:05 UTC | 48 | hub + 4 hosts | pi 1.4.0 | glm-5.3-flash:fast | 90 min | 333 of 333 | 47 agents stopped with their checks passed; 1 stopped when the Stop gate failed open. |
| 2026-10-31 16:56 UTC | 48 | hub + 3 hosts | pi 1.4.0 | glm-5.3-flash:fast | 90 min | 333 of 333 | All 48 agents stopped with their checks passed. |
| 2026-10-31 17:44 UTC | 48 | hub + 3 hosts | pi 1.4.0 | glm-5.3-flash:fast | 90 min | 0 of 333 | The agents' logs show no work, and main held 41 documents where complete runs held 64; nothing passed. |