Evidence
Greenfield Experiment 2 — construction with no method authority in the repository
The same frozen requirements, built in an empty repository that contained no architecture rules, no method documents and no conventions of any kind.
What this was
- Status
- Complete
- Repository
- kemiller2002/SDE-Engineering-Trial-2-Greenfield-Construction
- Branch
- claude/greenfield-engineering-experiment-2-ee44qz
- Baseline tag
- greenfield-experiment-2-baseline-20260907-090717
- Baseline commit
- 45c651ce2d26
- Ending commit
- 633192d1b60f
- Executed by
- Claude Code over the Claude Agent SDK, configured model claude-sonnet-5, single continuous remote session, no human intervention.
Objective
Build the full time-entry system from the frozen requirements package, recording preregistered predictions before construction and comparing them against what actually happened.
Hypothesis
Preregistered, written before any code existed and never revised afterwards: predicted file counts, source size, the highest-risk requirement area, and the number of repair cycles.
Method
The repository was tagged at baseline, the requirements read in full, and a preregistration frozen before any product code was written. Construction then proceeded through six checkpoints, T0 to T6, with telemetry recorded contemporaneously at each. Metrics the runtime does not expose to a running agent were marked unavailable rather than estimated.
Recorded metrics
58 / 58
Acceptance criteria implemented and traced
Self-reported
- Measures
- Acceptance criteria from the frozen requirements package with an implementation and a named test, mapped ID by ID.
- Collected by
- A traceability document mapping each criterion to the test covering it, written alongside the implementation.
- Does not show
- Traceability means every criterion has a test. It does not mean the tests are good: this build's own suite found zero new defects, and an independent reviewer of the replication found five things its equivalent suite missed.
Sourcekemiller2002/SDE-Engineering-Trial-2-Greenfield-Construction / experiment/REQUIREMENTS-TRACEABILITY.md at claude/greenfield-engineering-experiment-2-ee44qzprivate repository87 / 87
Tests passing
Instrumented
- Measures
- Tests across 12 files, all passing, none skipped, with a strict type check clean.
- Collected by
- Test runner output at the final verification checkpoint.
- Does not show
- All 87 passed on first run against the already-corrected implementation, finding zero new defects. The suite's demonstrated value here was specification and regression protection, not discovery.
Sourcekemiller2002/SDE-Engineering-Trial-2-Greenfield-Construction / experiment/TELEMETRY.md at claude/greenfield-engineering-experiment-2-ee44qz — T5 — Verified Implementationprivate repository2 + 1 + 1 + 0
Defects, by the mechanism that found each
Self-reported
- Measures
- Two found by the type checker, one by a manual smoke test against real disk, one by adversarial self-review, and zero by the automated test suite.
- Collected by
- Recorded per checkpoint as each was found, with the finding mechanism named.
- Does not show
- Four mechanisms, four different results. This is a description of one build's defect discovery, not a ranking of mechanisms: the self-review found one because it looked for a shape the tests could not express.
Sourcekemiller2002/SDE-Engineering-Trial-2-Greenfield-Construction / experiment/TELEMETRY.md at claude/greenfield-engineering-experiment-2-ee44qz — T5 — Verified Implementationprivate repository0
Runtime dependencies
Instrumented
- Measures
- Third-party packages the built application needs at runtime. Everything else is a development dependency.
- Collected by
- Read from the dependency manifest at experiment close.
- Does not show
- Zero runtime dependencies is a property of this problem — a business-rules engine behind a thin boundary — not a general outcome of the method.
Sourcekemiller2002/SDE-Engineering-Trial-2-Greenfield-Construction / experiment/TELEMETRY.md at claude/greenfield-engineering-experiment-2-ee44qz — T4 — Primary Implementation Completeprivate repository2,106 + 1,421 lines
Source and test size
Instrumented
- Measures
- 22 source files and 12 test files, 3,527 lines in total.
- Collected by
- Counted from the working tree at the final checkpoint.
- Does not show
- This exceeded the build's own preregistered estimate by roughly 25 to 45 percent. Line counts measure size, not quality, and the preregistration was wrong about them.
Sourcekemiller2002/SDE-Engineering-Trial-2-Greenfield-Construction / experiment/TELEMETRY.md at claude/greenfield-engineering-experiment-2-ee44qz — T4 — Primary Implementation Completeprivate repositoryNOT OBSERVABLE
Token and cost totals
Not observable
- Measures
- Token counts, cost and context occupancy for this run.
- Collected by
- Not collected: this runtime exposes none of them to a running session. Recorded as unavailable at every checkpoint, and again explicitly at experiment close.
- Does not show
- An absence of data, not a zero. No cost comparison with any other experiment on this site is possible.
Sourcekemiller2002/SDE-Engineering-Trial-2-Greenfield-Construction / experiment/TELEMETRY.md at claude/greenfield-engineering-experiment-2-ee44qz — T6 — Experiment Closedprivate repositoryFindings
- All 12 product requirements, all 20 business rules and all 58 acceptance criteria were implemented, with an ID-by-ID traceability mapping from each criterion to the test that covers it.
- All nine open questions in the requirements package were preserved as unresolved rather than guessed past.
- The final suite was 87 tests across 12 files, all passing, none skipped, with a strict type check clean at every checkpoint from T2 onward.
- The built system has zero runtime dependencies: everything beyond the language standard library is a development dependency.
- Defects were found by four different mechanisms, and each found a different thing: the type checker found two, a manual smoke test against real disk found one, and adversarial self-review found one that no test had caught.
- The preregistration correctly identified the highest-risk area — the reconciliation requirement — and was wrong about the specific failure inside it.
What this refuted
- The test suite itself found zero new defects. Every test passed on first run against the already-corrected implementation. The experiment records this plainly rather than presenting 87 passing tests as 87 defects prevented: the suite's demonstrated value here was specification and regression protection, not discovery.
- Actual repair cycles came in below the preregistered range of three to six: there were two.
- Source and test size together exceeded the preregistered estimate by roughly 25 to 45 percent.
Anomalies, failures and confounders
- The repository contained no engineering-method authority of any kind at baseline — no architecture rules, no construction method, no conventions. This is recorded as a genuine finding about the experiment's conditions, not as an oversight.
- The baseline tag could not be pushed: HTTP 403 on the tag ref across four retries. The local tag stands as the baseline record.
- A dependency vulnerability in the test runner's transitive dependencies was found and cleared before any domain code was written.
- Timer behaviour is verified only with injected timestamps. The real elapsed-time path is exercised by type checking and a sub-30-second integration test, not by an actual-elapsed-time test.
- Concurrency paths are exercised by fault injection and sequential calls, not by genuinely concurrent processes.
Limitations
- This is one agent's single pass, self-assessed at the end. No external audit was performed — a gap the replication was specifically designed to fill, and which it filled by finding four more defects.
- Token, cost and context figures are unavailable for the whole run: the runtime exposes none of them to a running session. They are marked unavailable rather than estimated, and no cost claim is made.
- The elapsed wall-clock figure is a session timestamp span, not active compute time, and includes substantial documentation writing interleaved with code.
- Nothing here compares this construction against an alternative method. It is a record of one build, not a controlled comparison.
Interpretation
An empty repository, a frozen requirements package, and no method authority of any kind. That
absence is not incidental to the result — it is the condition being reported. There were no
architecture rules to follow, no construction method in the tree, no conventions. Whatever
structure the build has, it chose.
What it chose is the thing worth noticing: a strictly one-directional dependency flow with a
pure domain that knows nothing about storage, effects returned as data, a single parameterised
validator, named predicates shared between the transitions and the capability projection, and
a four-outcome result type at the storage boundary including an explicit unknown. None of that
was prescribed. The replication, run independently, arrived at substantially the same set.
The most honest thing in this record is the note that the test suite found zero new defects.
Eighty-seven tests, all passing on first run against the already-corrected implementation. The
record states plainly that the suite's demonstrated value here was specification and regression
protection rather than discovery, and that the defects it would have caught were caught
earlier, by a manual smoke test, before the suite existed.
It would have been trivially easy to present 87 passing tests as 87 defects prevented. The
record declines to.
The self-review found something the tests could not: a validation check that could never fire,
with a test that passed for the wrong reason. That is a failure shape assertions are
structurally bad at — the test was asserting something true and irrelevant — and it took a
person reading adversarially to see it.
Its own preregistration was directionally right and specifically wrong: it named the
reconciliation area as the top risk, and the defects did concentrate there, and it did not
predict which ones. The replication's preregistration, written independently, did exactly the
same thing.