Results of the main run

27 September 2026, 21:09 to 21:21 CEST. 24 tasks, three ways each, 72 runs in a random order, one fresh session and one budget per run, model Claude Sonnet 5, total cost 4.04 USD. The specialists ran live on this domain. Route below means one of the three setups from the front page: A works alone, B may search the directory, C has the specialists named in its instructions.

Headline numbers

Success criteria, fixed before the run

• passed: B correct on at least 12 of 16 checking tasks (12 of 16)
• failed: B at least 4 checking tasks ahead of A (1 ahead)
• passed: B calls no specialist on at least 7 of 8 simple tasks (8 of 8)
• passed: B mean cost at most 150 percent of A (104 percent)
• passed: B median time at most 150 percent of A (101 percent)
• passed: B at most 1 checking task behind C (1 behind)

Five of six passed. The one that failed is the one the experiment was for: with the directory, the agent was not clearly better than alone.

What happened, per task type

The simple tasks (sorting, extracting, shortening a headline) were solved alike by all routes, 7 of 8 each; the one miss was a headline that all three cut to seven or eight words instead of six, a defect of the task.

The observation that matters

Route B was told it could search the directory and hire a specialist, and did so once in 24 tasks. Route C had the same specialists named in its instructions and delegated 11 times. The one directory lookup worked: it returned a reachable Link Auditor, the call succeeded, the task was solved. The directory was not broken, it was not consulted. Whether that is a matter of how the tool was described, how the task was phrased, or a sound judgement by the agent that it could do the work itself, is the next question, testable on the same 24 tasks with one change at a time.

Follow-up 1: does the tool description matter? (28 September)

Same 24 tasks, route B only, one change: the directory tool now names what the two specialists do. B searched on 6 of 24 tasks instead of 1, all six on link tasks, and scored the same 12 of 16 (1.37 USD, 24 runs). In this pair of runs the description changed how often the agent looked; the total did not move, although single tasks flipped both ways (one redirect task solved in the first pass failed in the second).

Follow-up 2: help the agent really needs (28 September)

A third specialist, Handbook QA, holds the internal operations handbook of Norvane Systems GmbH, a company invented for this test; the handbook is served by no page and indexed by no search engine. Twelve questions can be answered only from it, four control questions can be answered from adrianfoehl.com. Four predictions were written down before the run: A answers at most 2 of the 12 handbook questions; C at least 10; B searches the directory on at least 8 of the 12; on the four web questions no route delegates more than once and every route gets at least 3 of 4.

Route A invented nothing: all twelve answers were an explicit "could not be established", nine of them after a web search and three without one. In the first pass four B runs failed with HTTP 429 from this site's own per-address rate limit, which the experiment runner tripped; the limit was raised and route B re-run in full (16 runs, 0.99 USD), with the result above. Both passes are kept. All four predictions held: on these questions the agent searched when it could not answer, found the specialist, called it and answered correctly, and did not search when it could answer itself. Whether the final answer was built from the specialist's reply is not traced in the logs; the call and the correct answer are.

Follow-up 3: other models as the requesting agent (28 September)

Both test sets were repeated with Haiku 4.5 and Opus 5.5 as the requesting agent; the specialists, tools, budget and answer keys stayed the same. On the 16 checking tasks, Haiku scored A 8, B 9, C 10 and never searched the directory; Opus scored A 14, B 15, C 15, searched 0 of 24 times, and delegated 9 of 24 with the list in its instructions. On the twelve handbook questions, Haiku searched 3 times and scored B 2 of 12 (A 0, C 11); Opus searched 12 times and scored B 12 of 12 (A 0, C 12), with no delegation on the four web-answerable questions from B (C delegated once). Sonnet remains at B 12 of 12 with 12 searches. Added late the same evening: Qwen3 30B-A3B running locally through Ollama (setups B and C only, no web search available, prices zero, 32 runs) searched 12 times and scored B 12 of 12, C 12 of 12, traps 2 of 4 in both; one C run aborted after two malformed specialist calls. The prediction written before that run, that the local model would behave like Haiku, failed upward. The next morning: Qwen2.5 32B (Ollama) searched 12 of 12 and scored B 11 of 12, C 12 of 12, traps 2 of 4, no aborts; Qwen3.8 27B (LM Studio) searched 12 of 12 and scored B 12 of 12, C 11 of 12, traps 3 of 4 and 4 of 4, one C run aborted, median 63 seconds; and, from a harness built by a second coding agent, OpenAI Codex, working in the same repository (gpt-6-astra, effort low, 32 fresh Codex sessions with the experiment tools over MCP, ChatGPT subscription; its report scores nothing and notes that Codex system instructions and tool names differ from this harness), B searched 12 of 12 and, scored here against the key, got 12 of 12 with 3 of 4 traps, C 12 of 12 with 3 of 4, no trap delegations. Its first batch of 19 runs was interrupted when this experimenter moved that session's files for a deploy; the resumed batch covers all 32 pairs. On 30 September the Codex agent added GPT-6 Sol and GPT-6 Luna (48 runs each, setups A, B and C) and Astra's missing setup A (16 runs); scored here against the key: alone all three 0 of 12; with the directory Sol 12 of 12, Luna 9 of 12 (searched 12 of 12; two questions lost to a no_match because Luna requested source_support and handbook_qa in one query and the directory returns only agents with every requested skill, one lost after both allowed specialist calls went to the source checker); with the fixed list all three 12 of 12. After the directory was changed on 30 September to return agents with any requested skill (ranked by coverage), Luna ran setup B again (16 runs, prediction fixed before: at least 11 of 12, at most 1 of 4 web questions delegated): 11 of 12, 4 of 4 web questions answered, none delegated; three of its discover queries asked for two skills and now reached the handbook agent; the one miss asked only for source_support. Report: results/codex-luna-anymatch-2026-09-30T06-39-54-581Z. Traps 3 of 4 in every route. Report: reports/2026-09-30-openai-three-model-handbook-comparison.md. A detail found afterwards: the experiment runner builds route C's list from the local seed file, not from the live database, so in the Haiku and Opus handbook runs C's instructions also named the superseded 2026-2 agent (registered that afternoon for follow-up 4); C never called it. The Sonnet handbook run predates that agent. The stronger the model, the smaller the gap between the three routes on tasks it can do itself. Whether the directory gets used at all when help is really needed is a property of the requesting model: Sonnet and Opus looked every time, Haiku mostly did not, even though it could not answer alone.

The first Opus handbook pass was cut short after 32 of 48 runs by the API account's spending limit ("specified API usage limits") (16 runs failed at zero cost, scored on completed runs only: A 0 of 7, B 7 of 7, C 10 of 10) and was re-run in full once the limit was raised. Both passes are in the results files.

Follow-up 4: two providers, one outdated, listed first (28 September)

A public overview page of Norvane Systems on this site carries facts from the older handbook edition 2026-2 (eight outdated, four unchanged). A second specialist, Handbook QA (edition 2026-2, superseded), was registered and holds that older edition; the directory's sort rule (most recently checked first) placed it ahead of the current one. Twelve questions named the public page and asked for the current value. Predictions written before the run: A gives the outdated value on at least six of the eight changed facts; B and C each choose the 2026-3 agent on at least nine of twelve.

All three predictions held. Route A's one correct changed fact was a hedged "cannot establish" that happened to contain the right number; the other seven repeated the public page. Routes B and C never called the superseded agent, although B saw it listed first on every lookup (per-case provider ids in the noisy analysis file in the repository's reports folder). The caveat is in the data the agent saw: both the directory entry and the agent card of the older edition say "superseded" and name the newer edition. This test shows that the requesting agent reads and follows what a directory says about its entries; it does not show that an agent can detect a stale provider that does not announce itself. That is the next variant: same setup, neutral descriptions.

Follow-up 5: neutral labels (28 September)

Same two providers as in follow-up 4, 2026-2 sorted first, but the directory rows no longer carry "superseded" or "current": names "Handbook QA (edition 2026-2)" and "Handbook QA (edition 2026-3)", descriptions and limitations naming only the edition. Twelve handbook questions, routes B and C. Predictions: B and C each choose the 2026-3 agent on at least 9 of 12; if either calls 2026-2 on 3 or more, the honest label was the mechanism in follow-up 4. Result: B 12 of 12, C 12 of 12, both called the 2026-3 agent on all twelve questions and the 2026-2 agent never; no run called both. The edition number alone was enough. The agent cards, which still said superseded, were never fetched (no card fetch in any run of follow-ups 4 or 5).

Follow-up 6: changing supply, two epochs (28 September)

Route C's provider list was frozen to a file before each change ("fixed list"), route B asked the live directory. Epoch 1, the provider moves: the 2026-3 handbook agent got a new path, its old path answered 410 Gone, the directory row was updated. Predictions: C solves at most 2 of 12, B at least 11 of 12 with a directory lookup on every question. Epoch 2, a newer edition: a third edition 2026-4 (eight of the twelve answers changed, four unchanged) was registered as its own agent; the 2026-3 agent was removed from the directory but its endpoint kept answering; C's frozen list, taken after epoch 1, still named 2026-3 at its new address. Key: the 2026-4 values. Predictions: C gives at most 4 current values of 12, B at least 11.

Epoch 1: predictions held. Route C called the dead address, got 410, tried a second time on most questions, and either gave up or hit the run's two-call limit; it never reached the handbook. Route B looked the directory up on every question and called the new address. Epoch 2: predictions held. Route C called the 2026-3 agent on every question and got confident answers from the old text; its four current values are exactly the four unchanged facts. Route B looked the directory up on every question and called the 2026-4 agent. This is the first pair of runs in which B beats C, and it measures the directory's own claim: it follows a change that a written list misses. The changes were made by the experimenter on their own directory, with C's list frozen by design; a real list owner may update sooner.

Follow-up 7: foreign supply, a2aregistry.org (28 September)

The three specialists of this site are registered in a2aregistry.org (probe WORKING) since 28 September. In the other direction, a Sonnet 5 requester with two tools, keyword search over that register and an A2A call to any agent found (both protocol dialects, 0.3 and 1.0), no web tools and no wallet, ran twelve tasks picked from services the register lists. Predictions before the run: non-error replies on at most 8 of 12, usable answers on at most 6, at least 2 sales or sign-up replies, and at least 2 unmarked own-knowledge answers.

Predictions one to three held; the fourth failed in the good direction, every own-knowledge answer was marked. Fitting agents existed in the register for at least six tasks; keyword search surfaced them for two. A first pass (12 runs, 0.25 USD) is kept but invalid: the caller spoke only the 1.0 dialect and 264 of the 457 registered agents run 0.3, answering "method not found". Per-call replies are in the registry analysis file in the repository's reports folder.

Follow-up 8: selection without a cue, and with verifiable evidence (29 September, run by the Codex coding agent)

Two synthetic diagnostics by the Codex coding agent (GPT-6 Astra, low effort, twelve fresh sessions each, deterministic local helpers, no live calls, protocol and criterion frozen before the run, openly authored scorer, no blind grading). Diagnostic 1: three helpers with identical descriptions return a current, a stale and a fabricated value; six role permutations, each with an independent source reachable or answering 503. Source reachable: 6 of 6 read the source, no helper consulted, correct. Source down: 6 of 6 consulted all three, named the disagreement, abstained; 0 adopted the stale or fabricated value. Diagnostic 2: same roles, each answer carrying an Ed25519-signed, dated evidence envelope; the requesting agent has a verifier bound to one pinned issuer key that returns a boolean. Authentic current evidence present: 6 of 6 selected the correct helper, verified, correct. None present: 6 of 6 abstained with verified false. Zero unsupported verification claims; 36 signature checks per diagnostic, 18 positive and 18 negative. Reports and frozen protocols are in the repository (reports/2026-09-29-codex-selection-analysis.md, reports/2026-09-29-codex-evidence-selection-analysis.md). That session's own limits: the pinned issuer is a granted trust root, not a solved identity problem; authenticity is not truth in general.

Before the main run

On eight small development tasks (two or three links each) the agent alone was already perfect, and specialists added nothing. On eight harder development tasks (twelve links, ten anchors, five claims each) the agent alone solved 3 of 6 checking tasks and with the specialists listed 6 of 6, at equal cost. That gate was passed before the main run was built.

What the two tests say together

In this task set, the directory condition did not outperform the agent working alone when the agent could do the work itself, and the same agent used the directory on every task it could not do alone. Discovery worked whenever it was tried. Whether it is tried depends on the requesting model: a small model left the directory unused even when it could not answer. Given two providers with honest descriptions, the agent chose the current one every time. Under changing supply, the directory beat a frozen list: when the helper moved, B 12 of 12 against C 0 of 12; when a newer edition replaced it, B 12 against C 4. What remains open is how fast a real list owner would have updated, whether an agent detects a stale helper without any cue, and whether anyone would pay.

Limits

• Three models from one vendor, one narrow domain (links, anchors, sources on one website) plus one synthetic handbook, 16 checking tasks and 12 handbook questions. Small numbers; a difference of one task is noise.
• The tasks, the specialists and the answer key came from the same person and their AI tooling. The key was written by two AI models independently from the same raw data (110 of 116 verdicts agreed; the six disagreements, all in the two partly-supported tasks, count either answer as correct).
• Two things were changed after the run and are logged: a bug in the quote checker (it ignored markdown markers, wrongly failing five verbatim quotes of route B on one task) and one key line disputed by all three routes with a defensible quote, left as it was because it changes no criterion.
• The main run measures procedural specialisation with equal tools; only the handbook tests cover a specialist that holds something the requesting agent cannot reach.
• Every score on this site is mechanical: a verdict or answer matched against the key, quotes checked against the live page. No human has blind-graded the runs; the run reports still name human grading as the final word, and that has not happened.
• The main result is sensitive to the key. Under the stricter reading (first listed verdict only, no acceptance lists) it is A 11, B 11, C 12 of 16: B's one-task lead over A disappears.

Experiment by Adrian Föhl. Method, rules and the decision fork were fixed before the run; a blind second opinion from a model from another vendor reviewed the design. Write-up to follow at adrianfoehl.com.