Benchmark: AI reviewers finding planted bugs with fact tools or source

Table fields, contract cases, and pass/fail rules

Each round below renders one table, one row per reviewer run, then a verdict line. Here is a real row read out in full, the first row of round 008:

tooling-default-uncapped · tooling · default · 96.2% · 80.0% · +5 · 0 · 137.0k · 21.5 · 0.07%
A run named tooling-default-uncapped, which used the fact tools (mode tooling) on the frontier-tier model with no spend cap. Against the 46,922-line program it found 96.2% of the 26 planted defects (25 of 26) and 80.0% of the 5 defects tagged subtle, i.e. 4 of 5. It scored +5 on calibration (it never claimed a certainty the checker could not grant), made 0 false accusations, spent 136,989 tokens over 21.5 minutes, and read 0.07% of the program, about 33 lines out of 46,922.

Probe: the name of one run, written mode-model plus a variant tag where a round ran the same pair more than once (-uncapped and -capped60k in round 008, -budget in rounds 006 and 007). Mode: which reviewer this was. tooling is the probe briefed to work from the fact tools, and baseline is the probe that reads source. These two words are the main axis of the whole benchmark. Model: Anthropic Claude tiers as run by the coding harness. “default” is the harness's default (frontier-tier) model at the time of the round, and “sonnet” and “haiku” are the mid and small tiers. Rounds record the tier, not the exact model ID, so a round cannot be re-run under identical conditions. Recall: how many of the planted defects the reviewer found (a partially-correct answer counts half). Subtle: recall restricted to the defects designed to survive a first pass: buried behind indirection, spread across files, or invisible to static reading. “n/a” means the round planted none. Subtle denominators are small (2–5 defects), so one half-point moves the number a lot. Calibration: did the reviewer overclaim? Scored over the round's explicit verdict questions (“does this declared rule hold?”), not per defect: +1 for a correct verdict or a correct “can’t be sure, because…”; −1 for declaring safe what the ground-truth checker could not confirm. The number of scored questions varies by round, so compare within a round, not across rounds. A positive score is printed with a + and shown in green, a negative one with a − in red, and a zero is printed bare and uncoloured. The sign carries the meaning, and the colour only reinforces it. No round has yet produced a negative score. A zero means the round's verdict questions cancelled out or scored nothing. It says nothing bad about the reviewer. False +: findings the reviewer asserted that the answer key contradicts, i.e. wrong accusations. Lower is better, and 0 is clean. Tokens: the AI agent's total token usage, which is the cost of the review. Minutes: wall-clock time the run took. Recorded for context only. It plays no part in the pass/fail rule, and it is not comparable across rounds run on different hardware. % source read: lines of the generated program the reviewer read, over its total line count. Re-reads count, so a reviewer that revisits files can exceed 100%.

Planned budget, in the grey line under each round title from round 001 on (the pilot had none): the tokens budgeted for the round as a whole before it ran, kept so plan can be compared against actual. It is bookkeeping. It constrains no probe, and the rows beneath it routinely sum to less. Round 008 did test real caps, and reports honestly that they failed to bind.

What “contract accuracy” is, and why it is not Recall. The two accuracy figures in the verdict line are not scored over the round's whole defect set. An artifact-contract case is one small program added to a standing corpus, carrying a ground-truth label: is this binding's result fabricated, does this declared rule survive every interleaving, is this permission check real? Each round contributes two such cases, one drawn from each defect family, and the gate asks both reviewers to classify them; contract accuracy is the fraction they get right, so with two cases it can only ever read 0, 0.5 or 1. (Round 010 is the exception: it scores 14 cases, its whole conformance census.) Recall, by contrast, is scored over every planted defect. The two can and do disagree: rounds 006, 007 and 008 all pass with contract accuracy tied at 1.0 while the Recall column plainly shows the tooling probe finding fewer defects than the source reader. That is a real weakness of the gate. A two-case accuracy test is too coarse to detect the recall gap its own table displays. Read the verdict as “cheaper at no cost on the contract cases”, and read the Recall column for who found more.

Pass/fail: after applying the contract-case definition above, a round passes if the fact-using reviewer did not lose contract accuracy and was cheaper. Formally, that is contract accuracy Δ ≥ 0 and tokenLift above 1.000, where Δ is the fact reviewer's contract accuracy minus the source reader's, and tokenLift = source-reader tokens ÷ fact-reviewer tokens, so above 1 means the facts were cheaper. Each verdict compares exactly one pair of probes. Where a round ran several, it is the frontier-tier (“default”) pair. Round 008 ran two frontier-tier pairs, and its verdict uses the uncapped one. Rounds that ran a single pair are judged on that pair, whatever its tier: round 010's verdict is its sonnet pair. Round 005a is the one round with no verdict, because round 5's official comparison was made in round 005b.

Each round's raw scores JSON, linked below the round, itemizes the exact per-item scoring: every planted defect's found/missed status and every calibration event. Scoring is adjudicated by this project's own harness against the generator's answer key. Round numbering: round 009 is absent because iteration 9 only built tooling and ran no measurement round, so there was nothing to score.