What happened when we measured it

A passing test suite is not evidence the work is done

This is the strongest conclusion on the site, it is supported from two independent directions, and it is the one with the clearest operational consequence.

A fresh agent, with no knowledge of how a build had been constructed, reviewed a finished implementation that had already passed a 90-test suite, a strict type check with noUncheckedIndexedAccess and exactOptionalPropertyTypes, a dependency audit, an architecture check, and an adversarial self-review that included three mutation tests deliberately breaking shared guards to confirm they were load-bearing.

It found five defects. None had been caught by any of that. One was a result case declared in a type that no code path could construct — the type checker was satisfied, ninety tests were satisfied, and the contract was incomplete.

Independently, METR 2025 found the same shape in someone else's repositories: agent solutions passing reference tests and still failing merge-readiness review. Two research programmes, different codebases, same finding.

Limit. One verification pass on one codebase. It shows these mechanisms missed what that reviewer found; it does not quantify how much any mechanism generally misses. Every agent involved came from one model family, so a shared blind spot cannot be ruled out.

Conventional code goes quiet exactly where it matters most

The same domain was rebuilt under a deliberately ordinary persona — a competent professional developer reaching for mutable classes and bare primitives by default. Not a strawman, and the distinction matters: the finding is about type design, not carelessness.

When the unit of measure changed, renaming the duration field caught eight call sites in the compiler. Around fourteen more passed duration as a bare positional integer — where the literal 10 had meant ten six-minute units, sixty minutes, and now meant ten minutes. Identical source text, six times the duration, no compile error. The build was clean and all thirty tests passed while that was unresolved.

In the JavaScript rebuild, the language caught nothing at all, at any point, for any reason. A property rename across two production files produced zero errors, zero warnings and zero crashes. The program ran.

Against that, the state-structured builds: every ripple location surfaced all of themRipple sites found by the compiler rather than by searching, and in the F# trial 17Ripple sites surfaced in a single build across two root causes — each root cause showing every affected call site at once. Zero required a manual search.

Limit. Comparing the state trials against the conventional rebuilds is partly a comparison of persona, and the state trials' own reports say so. What is not a persona effect is the structural point below.

The difference, stated as numbers

The two tables below are the whole comparison. Everything else on this page is context for them. Every figure links to its own record — definition, how it was measured, what it does not show, and the file and section to check it against.

1. The same intended change, propagated four ways

The unit of measure changed: recording became exact to the minute, six-minute rounding moved to a derived billing view. The identical change was applied to the same domain in four implementations. The question is how many of the affected call sites the machine named, and how many a person had to find.

The C# row is the one to read carefully. Eight of roughly twenty-two sites were direct member accesses the compiler reported. The other fourteen passed duration as a bare positional integer, so the literal 10 went from meaning sixty minutes to meaning ten — identical source text, six times the duration, clean build, all tests passing. Two of those were correct by coincidence (2 of them). In the JavaScript rebuild the language named nothing at all; 8, from one root cause test failures caught the change instead, and only because those assertions compare numbers rather than reference the renamed property.

The two Ordo rows show no manual searching for this change. As whole-trial context rather than a cell in the table above: the C# state-system trial ran 18 repository searches in total, for every purpose combined, with 0 context expansions beyond its planned reading list.

2. Mistakes, by the mechanism that caught each one

This is where the popular version of the claim breaks. Raw mistake counts do not favour more static checking — the build with the least of it scored best. What differs is which mechanism did the catching.

What the two tables together support, and nothing more: the conventional C# rebuild made 6 mistakes and the JavaScript one 2, at comparable size. Of the C# six, 3 of 6 were invisible to every mechanism and were caught only because someone deliberately re-read a diff before running it. Swapped string arguments, a stale revision inside a loop, a signature change hidden behind var — none of these is catchable by a compiler even in principle. In the state-structured builds the equivalent substitution is unwritable, because a duration and a time are different types. That is the claim. It is a claim about which failures remain possible, not about how many occur.

Why this table has two columns and not four. The two state-system trials did not log mistakes in this format — their records classify defects by class and report 0Runtime domain defects and 0Runtime domain defects runtime domain defects respectively, which is not the same measurement. Rather than convert between formats and present the result as like-for-like, the columns are left absent. Table 1 is the four-way comparison, because propagation was recorded the same way in all four.

Where the machine caught nothing

Five measured cases, all from experiments designed to find them. These are the boundaries of the claim above, and they are listed with numbers for the same reason the numbers above are: so the limit can be checked rather than taken on trust.

And the one that cuts hardest, because it runs against the direction the method predicts: in the third controlled experiment, as-experienced mechanical discovery went 42.1% to 31.6% — lower after hardening, not higher — across 19 and 19 required change sites, while search operations fell 38 to 14 and repair loops 2 to 1. Discovery rate and engineering cost moved in opposite directions. The site does not have a resolved explanation for that, and the hypothesis it belongs to is open.

Some things are not measured — they are guaranteed

These are properties of the code, verifiable by reading it. They carry no confidence interval because they are not statistical claims.

Specification clarity is reachable, and it is the new bottleneck

arXiv 2605.01160 2026 concludes that under specification governance the constraint on velocity moves from generation to specification clarity. Two independent builds of the same frozen requirements tested whether that clarity is actually achievable by an agent.

Both builds implemented all 12 product requirements, all 20 business rules and all 58 acceptance criteria, with an ID-by-ID mapping from each criterion to the test covering it. Both preserved all nine open questions as unresolved rather than guessing past them. Neither had seen the other's work.

They converged on the same seven entities, the same three state machines, the same transition set, a mathematically identical overlap formula and an identical billing formula — and independently arrived at the same architecture for deriving capabilities from the predicates the transitions enforce, documenting the reason in nearly the same words. Across 41 requirement-level decisions they differed on one, and that one was a scope question about what to build rather than a disagreement about a rule.

Cold-start reconstruction was also tested directly: 10 of 10Cold-start questions answerable without opening source — what the application does, where authoritative state lives, which transitions are legal, where each workflow is implemented and how it is tested, all answerable from the generated documents without opening a source file.

Limit. Two builds are a replication, not a sample, and both came from the same model family — convergence between similar systems is weaker evidence than convergence between different ones. The cold-start figure was self-assessed after a documentation gap had been closed, not tested on an actual fresh reader.

Requirements-first construction held up

Recovering the requirements, modelling the states, and only then implementing produced a first pass with no runtime domain defects and nothing rewritten because earlier work was wrong. All sixteen domain files touched contained semantic decisions; none contained only mechanical propagation. Eighteen searches across the entire trial, with no occasion where in-hand context proved insufficient.

Limit, and it is a real one. This was a single uninterrupted pass with no later requirement change to force rework. The trial's own report is explicit that Semantic Change Coverage is therefore not applicable rather than zero, and that a fair claim about reduced rework would need a follow-up change-task trial it did not run. We are not making that claim.

The research behind it

Independent work establishing the problem, with what each source does and does not support:

Our own trials, with objective, method, findings, refutations, anomalies and limits recorded per experiment: the experiment records → and the full map of what is supported, provisional, contradicted and open →