The best-evidenced idea on this site, and the one that most often gets read as "write more
tests". It is not that.
In plain language
Each way of checking software is blind to something. A type checker cannot see wrong
behaviour. A test suite cannot see what its author did not think to assert. Neither can see a
declared outcome that no code path produces.
Heterogeneous verification means deliberately assembling mechanisms whose blind spots do not
overlap — and noticing that adding more of a mechanism you already have moves nothing.
Each row's right-hand column is what some later mechanism has to cover. This is why Ordo asks for several mechanisms with different sensitivities rather than a higher number from one.
The evidence
Four independent observations, in three codebases, all pointing the same way.
0 of 3
Checks that objected to a deliberately removed boundary arm
Instrumented
Measures
Of the compiler, the existing test suite and the running application, how many produced any signal when a required fan-out arm was removed.
Collected by
A deliberate probe: remove the arm, build, run the tests, query the database directly, then restore.
Does not show
One probe, one arm, one architecture. It establishes that this failure is real and reproducible here, not how often it occurs in general.
Mechanical checks that caught a present-but-inert boundary arm
Instrumented
Measures
Of the compiler, the architecture check, the schema-agreement test and the field-vocabulary agreement test, how many detected a boundary arm that existed but did nothing.
Collected by
A deliberate out-of-scope mutation re-creating Experiment 1's failure shape against the hardened architecture, with all four checks run against it.
Does not show
Exhaustiveness checking guarantees the arm exists. It says nothing about whether the arm does anything. A live integration test detected it; nothing else did.
Two found by the type checker, one by a manual smoke test against real disk, one by adversarial self-review, and zero by the automated test suite.
Collected by
Recorded per checkpoint as each was found, with the finding mechanism named.
Does not show
Four mechanisms, four different results. This is a description of one build's defect discovery, not a ranking of mechanisms: the self-review found one because it looked for a shape the tests could not express.
Independent-verification findings the builder's own checks had already caught
Instrumented
Measures
Of five findings from a fresh agent with no knowledge of the construction, how many were already covered by the builder's own 90-test suite or its own adversarial self-review.
Collected by
An overlap analysis performed against the existing suite and self-review document after the verification findings were received.
Does not show
One verification pass on one codebase. It shows these mechanisms missed what that one found; it does not quantify how much any mechanism generally misses, and a different reviewer would have found a different set.
The last of those is the sharpest. A fresh agent, with no knowledge of the construction,
reviewed a build that had already passed a 90-test suite, a strict type check, a dependency
audit, an architecture check, and an adversarial self-review that included three mutation
tests deliberately breaking shared guards to confirm they were load-bearing.
It found five things. None of them had been caught by any of that.
7, of which 4Defects found, and how many needed the extra pass of that build's recorded defects exist only because the extra pass
was run. Its counterpart build, which had no equivalent phase, was then checked directly —
and one of the same defect classes was found sitting in it, unaddressed.
The honest reading of the defect counts
One build recorded seven defects and the other two. That is not a quality comparison, and
the comparison document refuses to present it as one. It is largely a measure of how many
independent verification passes each build received. Stop the seven-defect build at the
point the two-defect build stopped, and it recorded three.
A defect count without its discovery protocol attached is not a measurement of anything.
What this is evidence for, precisely
That these mechanisms missed what that reviewer found, on that codebase. It is one pass, on
one build, and a different reviewer would have found a different set. It does not quantify
how much any mechanism generally misses.
There is also a shared-blind-spot caveat that cannot be ruled out: every agent in every
experiment here came from the same model family. Two systems that fail the same way are not
two mechanisms, however independently they are run — and this evidence base cannot currently
distinguish those cases. The proposition is classified RECOMMENDED and supported
(CL-020) with that caveat attached to it.
What an agent should do differently
Before declaring work verified, name the mechanisms that ran and what each one cannot
catch. If they all have the same blind spot, one of them ran.
Prove a shared predicate is shared: break it on purpose and confirm that everything
depending on it breaks too. If only one caller fails, the sharing was nominal.
Check that every outcome your types declare has a code path that constructs it. An
unreachable outcome is a contract you do not implement.
Run at least one check against the real boundary. The semantic no-op has evaded every
mechanical check in every trial in this programme, and integration testing has caught it
in every one.
How this is most often got wrong
Coverage read as assurance. One build's 87 tests all passed on first run and
found zero new defects. They were valuable — as specification, and as regression protection.
They were not what found the bugs.
Self-review counted as an independent mechanism. It is a good mechanism and
it found real defects in both builds here. It is not independent, and the zero-overlap result
is what that costs.
More tests as the answer to a missed defect. Sometimes correct. But if the
defect was of a shape tests cannot express, the next one of that shape will also be missed.