Results of the main run
27 September 2026, 21:09 to 21:21 CEST. 24 tasks, three ways each, 72 runs in a random order, one fresh session and one budget per run, model Claude Sonnet 5, total cost 4.04 USD. The specialists ran live on this domain.
Headline numbers
| route | checking tasks | simple tasks | delegated | mean cost | median time |
|---|---|---|---|---|---|
| A alone | 11 of 16 | 7 of 8 | 0 of 24 | 0.056 USD | 10.3 s |
| B with directory | 12 of 16 | 7 of 8 | 1 of 24 | 0.058 USD | 10.5 s |
| C with fixed list | 13 of 16 | 7 of 8 | 11 of 24 | 0.055 USD | 11.2 s |
Success criteria, fixed before the run
- passed: B correct on at least 12 of 16 checking tasks (12 of 16)
- failed: B at least 4 checking tasks ahead of A (1 ahead)
- passed: B calls no specialist on at least 7 of 8 simple tasks (8 of 8)
- passed: B mean cost at most 150 percent of A (104 percent)
- passed: B median time at most 150 percent of A (101 percent)
- passed: B at most 1 checking task behind C (1 behind)
Five of six passed. The one that failed is the one the experiment was for: with the directory, the agent was not clearly better than alone.
What happened, per task type
| task type | A | B | C | note |
|---|---|---|---|---|
| Valid links (2 tasks, 10 links each) | 2 | 2 | 2 | |
| Broken links (2 tasks, 10 links each) | 2 | 2 | 2 | |
| Redirects with expected target (2 tasks, 8 each) | 2 | 1 | 2 | B accepted an http page for an expected https target |
| Page anchors (2 tasks, 10 each) | 2 | 2 | 2 | |
| Source supports the claim (2 tasks, 5 claims each) | 1 | 2 | 2 | A called a supported number unsupported |
| Source supports it only partly (2 tasks) | 0 | 1 | 1 | the hardest category for every route |
| Source contradicts the claim (2 tasks) | 1 | 1 | 1 | one claim was answered contradicted by all three routes against the key; recorded as a key dispute |
| Evidence absent (1 task) | 1 | 1 | 1 | |
| Source not readable (1 task) | 0 | 0 | 0 | all routes read a login page as readable; the key says unverifiable |
The simple tasks (sorting, extracting, shortening a headline) were solved alike by all routes, 7 of 8 each; the one miss was a headline that all three cut to seven or eight words instead of six, a defect of the task.
The observation that matters
Route B was told it could search the directory and hire a specialist, and did so once in 24 tasks. Route C had the same specialists named in its instructions and delegated 11 times. The one directory lookup worked: it returned a reachable Link Auditor, the call succeeded, the task was solved. The directory was not broken, it was not consulted. Whether that is a matter of how the tool was described, how the task was phrased, or a sound judgement by the agent that it could do the work itself, is the next question, testable on the same 24 tasks with one change at a time.
Before the main run
On eight small development tasks (two or three links each) the agent alone was already perfect, and specialists added nothing. On eight harder development tasks (twelve links, ten anchors, five claims each) the agent alone solved 3 of 6 checking tasks and with the specialists listed 6 of 6, at equal cost. That gate was passed before the main run was built.
Limits
- One model, one narrow domain (links, anchors, sources on one website), 16 checking tasks. Small numbers; a difference of one task is noise.
- The tasks, the specialists and the answer key came from the same person and their AI tooling. The key was written by two AI models independently from the same raw data (110 of 116 verdicts agreed; the six disagreements, all in the two partly-supported tasks, count either answer as correct).
- Two things were changed after the run and are logged: a bug in the quote checker (it ignored markdown markers, wrongly failing five verbatim quotes of route B on one task) and one key line disputed by all three routes with a defensible quote, left as it was because it changes no criterion.
- The test measures procedural specialisation with equal tools. It says nothing about specialists that hold something the requesting agent cannot reach.
Experiment by Adrian Föhl. Method, rules and the decision fork were fixed before the run; a blind second opinion from a model from another vendor reviewed the design. Write-up to follow at adrianfoehl.com.