Evidence
Greenfield Experiment 2B — independent replication and convergence comparison
A second agent built the same requirements without seeing the first build, added an independent verification phase, and only then compared the two. The strongest single result on this site comes from that phase.
What this was
- Status
- Complete
- Repository
- kemiller2002/SDE-Engineering-Trial-2-Greenfield-Construction
- Branch
- claude/greenfield-experiment-2b-replication-n8pgtq
- Baseline commit
- d75a07747e41
- Ending commit
- fa73cf1f9454
- Executed by
- Claude Code, configured model claude-sonnet-5, single continuous session. One general-purpose subagent was spawned for the independent verification phase.
Objective
Replicate Greenfield Experiment 2 independently, add a verification mechanism the original did not have, and then — only after freezing its own results — compare the two builds to see what two independent agents converge on and what they do not.
Hypothesis
Six preregistered hypotheses, including that failures migrate to boundaries, that heterogeneous verification mechanisms find defects earlier ones miss, and that two independent builds converge more on semantics than on physical structure.
Method
Construction, verification and self-review were completed and frozen before the first build was opened at all. Self-review included three targeted mutation tests that deliberately broke shared guards to confirm they were load-bearing. A genuinely fresh agent, with no knowledge of the construction, was then spawned to review the result. Only after its findings were handled was the other build read, its test suite run, and a per-dimension convergence comparison written.
Recorded metrics
0 of 5
Independent-verification findings the builder's own checks had already caught
Instrumented
- Measures
- Of five findings from a fresh agent with no knowledge of the construction, how many were already covered by the builder's own 90-test suite or its own adversarial self-review.
- Collected by
- An overlap analysis performed against the existing suite and self-review document after the verification findings were received.
- Does not show
- One verification pass on one codebase. It shows these mechanisms missed what that one found; it does not quantify how much any mechanism generally misses, and a different reviewer would have found a different set.
Sourcekemiller2002/SDE-Engineering-Trial-2-Greenfield-Construction / experiment-2b/INDEPENDENT-VERIFICATION.md at claude/greenfield-experiment-2b-replication-n8pgtqprivate repository7, of which 4
Defects found, and how many needed the extra pass
Self-reported
- Measures
- Seven recorded defects. Three were found by the builder's own construction, testing and self-review; four were found only by the independent verification phase.
- Collected by
- Every defect logged with a stable id, its discovery mechanism, root cause, repair and regression evidence.
- Does not show
- A higher defect count here does not mean a worse build. It largely measures how many independent verification passes this build received compared with the one it is being read against.
Sourcekemiller2002/SDE-Engineering-Trial-2-Greenfield-Construction / experiment-2b/DEFECT-REGISTER.md at claude/greenfield-experiment-2b-replication-n8pgtqprivate repository90 / 90
Tests passing after verification
Instrumented
- Measures
- Tests across six files after the six regression tests added in response to the independent verification findings.
- Collected by
- Test runner output at the final checkpoint.
- Does not show
- Eighty-four of these tests existed before the independent review and did not catch any of the five things it found.
Sourcekemiller2002/SDE-Engineering-Trial-2-Greenfield-Construction / experiment-2b/TELEMETRY.md at claude/greenfield-experiment-2b-replication-n8pgtq — Engineering executionprivate repository35 tool calls, 164,383 tokens, 8.1 min
Independent verifier's own run
Instrumented
- Measures
- Measured cost of the verification subagent's run — the one point in the session where the tooling exposed usage for a unit of work.
- Collected by
- Reported by the subagent's own task-completion metadata.
- Does not show
- This covers the verification pass only. The main session's equivalent figures are unavailable, so this cannot be read as a proportion of the experiment's total cost.
Sourcekemiller2002/SDE-Engineering-Trial-2-Greenfield-Construction / experiment-2b/TELEMETRY.md at claude/greenfield-experiment-2b-replication-n8pgtq — Agent / modelprivate repository22 and 22
Source files in each of two independent builds
Instrumented
- Measures
- Source file counts from two agents building the same frozen requirements without seeing each other's work.
- Collected by
- Counted directly on both working trees during the comparison.
- Does not show
- The comparison warns explicitly against reading physical similarity as semantic convergence: the identical count masks real differences in how each build maps semantic clusters onto files.
Sourcekemiller2002/SDE-Engineering-Trial-2-Greenfield-Construction / experiment-2b/ARCHITECTURAL-CONVERGENCE.md at claude/greenfield-experiment-2b-replication-n8pgtq — C. Physical architecture convergenceprivate repository1 of 41
Requirements-scope divergences between two independent builds
Derived
- Measures
- Across 12 product requirements, 20 business rules and 9 open questions, how many were handled differently by the two builds.
- Collected by
- A per-dimension comparison of both codebases and both requirements inventories, performed after this build's own results were frozen.
- Does not show
- Two builds by the same model family with the same tooling. Convergence between similar systems is weaker evidence than convergence between genuinely different ones, and two runs are not a sample.
Sourcekemiller2002/SDE-Engineering-Trial-2-Greenfield-Construction / experiment-2b/ARCHITECTURAL-CONVERGENCE.md at claude/greenfield-experiment-2b-replication-n8pgtq — G. Convergence scorecardprivate repositoryFindings
- Not one of the five independent-verification findings was already covered by the builder's own 90-test suite or by its own adversarial self-review. The overlap was zero.
- Four of the seven recorded defects exist in the record only because that verification phase was run. Without it the count would have been three.
- One of those four was a declared outcome that could never actually be constructed: the reconciliation function had a Conflict result in its type that no code path could produce. The type checker was satisfied, the tests were satisfied, and the contract was incomplete.
- All 12 requirements, 20 business rules and 58 acceptance criteria were implemented and tested, with all nine open questions preserved unresolved — matching the other build exactly.
- The two independent builds converged strongly on semantics: identical entity sets, identical state machines, identical transition sets, and mathematically identical overlap and billing formulas, arrived at separately.
- Both builds independently arrived at the same architecture for deriving what an actor may do next from the same predicates the transitions use — never a second, independently maintained copy — and documented the reason in nearly the same words.
- Both preregistrations independently ranked the same requirement as the highest risk, and both were wrong about the specific failure within it. The risk ranking converged even where the defects did not.
What this refuted
- The raw defect counts between the two builds are not a quality comparison, and the comparison says so explicitly: the difference is largely a measure of how many independent verification passes each build received, not of how good either implementation was.
- One defect class the verification phase fixed in this build was then found, unaddressed, in the other build — direct evidence that the class was reachable by both independent constructions and was exposed only by the extra mechanism.
- Two builds produced exactly the same test line count from very different file structures. The comparison reports this as almost certainly coincidence and explicitly declines to use it as evidence for anything.
- One hypothesis — that greenfield scope is more predictable than legacy scope — was recorded as not testable from this run, because no legacy comparison exists. It is left open rather than answered.
Anomalies, failures and confounders
- The two builds diverged on one genuine requirements-scope question: whether any provisioning capability belongs in the system at all. Both left the underlying question open; they differ on what to build in the meantime.
- The two builds model illegal states differently. One makes an inconsistent entry impossible to construct at the type level; the other prevents it by construction discipline plus a load-time validator, and says so in its own self-review unprompted.
- Four acceptance tests failed on their first run. The cause was a defect in a test helper, not in the product code.
- Only five black-box scenarios were run live against both systems. That is a spot check corroborating the code reading, not exhaustive behavioural coverage of all 58 criteria.
- Tag pushes failed with HTTP 403 throughout, consistently. Branch pushes worked.
- The two experiments ran under different protocols, so their defect counts are not directly comparable without accounting for that asymmetry.
Limitations
- Two builds are a replication, not a sample. Nothing here establishes how often two independent agents converge in general.
- The zero-overlap result is one verification pass on one codebase. It shows that these mechanisms missed what that one found; it does not quantify how much any mechanism generally misses.
- Both builds were produced by the same model family with the same tooling. Convergence between two runs of similar systems is weaker evidence than convergence between genuinely different ones.
- Token, cost and context figures are unavailable for the main session. Only the verification subagent's own run was measured, because that is the one point where the tooling exposed it.
- The convergence comparison was written by one of the two participants. It reads the other build's code and runs its tests directly, but it is not a third-party comparison.
- No counterfactual arm exists: nothing compares either build against unconstrained development of the same requirements.
Interpretation
Two results here, and the second one is the reason this experiment matters more than its
replication status suggests.
The convergence
Two agents, the same frozen requirements, no contact between them until one had frozen its own
results entirely. They converged strongly on semantics: the same seven entities, the same three
state machines with the same transitions, a mathematically identical overlap formula, an
identical billing formula, the same four-outcome storage result, and the same architecture for
deriving capabilities from the predicates the transitions enforce. Both documented the reason
for that last one in nearly the same words.
They converged much less on physical structure — different file counts under most directories,
two different shapes of dependency injection, one splitting its application layer five ways
where the other split it two. The comparison is explicit that identical source-file counts are
the least meaningful thing in it, and that the two builds' exactly-equal test line counts are
almost certainly coincidence and are not used as evidence for anything.
Declining to use a striking number is a harder discipline than finding one.
The verification result
A fresh agent, with no knowledge of the construction, reviewed a build that had passed a
90-test suite, a strict type check, a dependency audit, an architecture check, and an
adversarial self-review including three mutation tests that deliberately broke shared guards to
confirm they were load-bearing.
Five findings. Zero overlap with any of it.
One was a result case declared in a type that no code path could construct. One was a tie-break
favouring the wrong side of an exact timestamp equality. One was a pair of diagnostic
categories that existed in the type system and were never actually surfaced. A fourth was dead
code. The fifth was not a defect at all but a genuine interpretation ambiguity, which was
recorded and deliberately left unresolved rather than settled by changing the product to match
a guess.
The comparison's own treatment of the defect counts is the part worth carrying away. This build
recorded seven defects against the other's two, and the comparison refuses to read that as a
quality difference, because four of the seven exist only because a verification pass was run
that the other build's protocol never included. Cut this build's process off where the other's
ended and it recorded three.
Then it checked whether one of its own defect classes existed in the other build. It did,
unaddressed — not because that build was careless, but because nothing in its protocol was
shaped to find it.