Evidence
The verified-program benchmark
Does an agent writing Bosatsu and checking it with our tools reach a program that passes a hidden harness on fewer tokens than one writing TypeScript from scratch? Four forum features, seventeen properties, and a harness neither arm sees decide.
The recorded result (iteration 001, 2026-09-05): the gate fails on every pair
Three fresh sonnet agents per arm per task, twenty-four runs. The
Bosatsu arm established 16 of 17 properties on every run;
the TypeScript arm established 16, 17, and 17. The Bosatsu
arm spent roughly twice the tokens: tokenLift
(TypeScript tokens per Bosatsu token) 0.54, 0.50, and 0.44
on the three run pairs. The pre-registered gate is accuracy(A) ≥
accuracy(B) and lift above 1 on each pair; it fails on all three, and the
number is published as it came.
| pair | verdict | accuracy, Bosatsu + tools | accuracy, TypeScript | tokens, Bosatsu | tokens, TypeScript | tokenLift |
|---|---|---|---|---|---|---|
| run 1 | fail | 0.941 | 0.941 | 527,348 | 286,849 | 0.54 |
| run 2 | fail | 0.941 | 1.000 | 482,343 | 242,753 | 0.50 |
| run 3 | fail | 0.941 | 1.000 | 540,819 | 238,418 | 0.44 |
Both misses are on the concurrent-reply task; the record attributes them to incomplete evidence, not an observed failing run.
summary.json carries
every run's established count, tokens, per-task results, iterations,
calibration, and the three gate verdicts;
gate-run1.json,
gate-run2.json, and
gate-run3.json
are the gate's own artifacts.
iteration-001.md
is the record: the design written before the runs, the results appended.
Ask your agent about anything here; it has the detail.
Where this fits
What this is about: Why an Analyzable Language?