Evidence
Benchmark: AI reviewers finding planted bugs with fact tools or source
Do machine-checked program facts help an AI reviewer find planted defects faster or more accurately than reading source? Each round plants defects in a generated Bosatsu program, from 2,028 to 46,922 lines across eleven rounds; one fresh reviewer is briefed to work from the fact tools, yichus's static-analysis queries, and one reads the source. This is a self-run benchmark: the same project builds the tools and the challenges, and each cell is one run.
Table fields, contract cases, and pass/fail rules
tooling-default-uncapped · tooling · default ·
96.2% · 80.0% · +5 · 0 · 137.0k ·
21.5 · 0.07%
A run named tooling-default-uncapped, which used the
fact tools (mode tooling) on the
frontier-tier model with no spend cap. Against the
46,922-line program it found 96.2% of the 26 planted
defects (25 of 26) and 80.0% of the 5
defects tagged subtle, i.e. 4 of 5. It scored +5 on
calibration (it never claimed a certainty the checker could not
grant), made 0 false accusations, spent
136,989 tokens over 21.5 minutes,
and read 0.07% of the program, about 33 lines
out of 46,922.
Probe names one run. Mode is tooling (briefed to use the fact tools) or baseline (reads source). Model is a harness tier, not an exact model ID. Recall is the share of planted defects found; Subtle is recall on the defects built to survive a first pass (n/a when none were planted). Calibration scores overclaiming on the round’s verdict questions. False + counts wrong accusations. Tokens and Minutes are the run’s cost and wall-clock. % source read is the share of program lines the reviewer read.
The gate’s contract accuracy is not Recall: it scores two small cases with a ground-truth label per round, which is too coarse to see the recall gap the table shows, a real weakness of the gate.
Pass/fail: a round passes when the fact reviewer kept contract accuracy and was cheaper (tokenLift above 1). There is no round 009: that iteration only built tooling.
Recorded results from committed artifacts
Best result under nominal caps that did not bind: on a 46,922-line codebase with 26 planted defects, an AI reviewer working from the fact tools found 25 of 26, and a reviewer reading source found 23. (Given an unlimited budget, that same source reader found all 26, at 1.85× the cost. Round 008 has both figures.)
The record so far: 6 of 10 judged rounds passed the gate, 4 rounds failed, and 1 round carries no verdict. Of the failures, all lost on cost, and 1 round lost on contract accuracy (1 on both). The gate does not measure defects found. Counting by those instead, the source reader matched or beat the fact tools in 8 rounds of 11, comparing the same pair each verdict was computed from. The tools' advantage is cost and scale, not raw finding power.
Early UI fixtures and experiments live in the archive.
Ask your agent about anything here; it has the detail.
Where this fits
What this is about: Fresh-agent testing