Evidence
Evidence
eleven experiments, four codebases, and every number on this site traceable to the artifact that produced it. Including the ones that came out badly.
The experiments
| Experiment | Status | Repository | Metrics | What it examined |
|---|---|---|---|---|
| Controlled Experiment 1 — boundary amplification on an unhardened architecture | Complete | kemiller2002/helix-note-application | 5 | How much of a new domain case's required propagation work a compiler can find on its own, and what happens to the part it cannot. |
| Controlled Experiment 2 — boundary authority hardening | Complete | kemiller2002/helix-note-application | 3 | Whether the blind spots Experiment 1 found can be closed mechanically, and what closing them does and does not change. |
| Controlled Experiment 3 — agent cost under hardening (corrected) | Complete — this record supersedes an earlier uncorrected draft | kemiller2002/helix-note-application | 8 | A paired A/B trial in which two isolated agents implemented the same frozen mission against the baseline and hardened architectures. The result that most complicates the programme's own thesis. |
| Greenfield Trial 1 — instrumented construction from frozen requirements | Complete | kemiller2002/SDE-Engineering-Trial-1-Greenfield-Construction | 6 | A time-entry system built from a frozen requirements package, with development-metrics instrumentation established before any implementation code was written. |
| Greenfield Experiment 2 — construction with no method authority in the repository | Complete | kemiller2002/SDE-Engineering-Trial-2-Greenfield-Construction | 6 | The same frozen requirements, built in an empty repository that contained no architecture rules, no method documents and no conventions of any kind. |
| Greenfield Experiment 2B — independent replication and convergence comparison | Complete | kemiller2002/SDE-Engineering-Trial-2-Greenfield-Construction | 6 | A second agent built the same requirements without seeing the first build, added an independent verification phase, and only then compared the two. The strongest single result on this site comes from that phase. |
| C# state-system trial — the same domain, structured explicitly | Complete | kemiller2002/time-entry-state-machine | 6 | The full time-entry domain built in C# against an explicit state model, reaching a first clean pass with no domain defects and no rework. |
| F# state-system trial — same requirements, same model, different language | Complete | kemiller2002/time-entry-state-machine | 2 | The C# trial's requirements and state model reused verbatim and implemented in F#, holding the specification fixed so the comparison isolates language. |
| Conventional C# rebuild — ordinary practice, requirement by requirement | Complete | kemiller2002/time-entry-state-machine | 8 | The same domain rebuilt the way a great deal of production C# is actually written, with every mistake recorded live and classified by what caught it. |
| Conventional JavaScript rebuild — the same domain with no type checker at all | Complete | kemiller2002/time-entry-state-machine | 7 | The identical requirement set built a third time in JavaScript, to see what changes when the language offers no static checking whatsoever. |
| Controlled baseline — everything held constant except explicit domain types | Complete — and it refuted the hypothesis it was built to test | kemiller2002/time-entry-state-machine | 3 | The most tightly controlled comparison in this evidence base, and the one that contradicts the easy conclusion. It is published because it did. |
How to read these pages
Each experiment page separates five things that a summary would merge: what the experiment found, what it refuted — including its own predictions — what went wrong during it, what its numbers are, and what a reader must not conclude from any of it.
Every metric card carries four things beyond its value: what it measures, how it was collected, what it does not show, and a link to the artifact. Following that link is the point of the exercise.
How each number was produced
Instrumented
A machine recorded it
A test runner, a build wrapper, a git count, a platform usage API. The most reliable class here, and still only as good as what the instrument was pointed at.
Self-reported
The agent recorded its own work
Search counts, discovery classifications, mistake counts. In the paired trial every self-reported build and test result was independently re-verified against committed diffs, with zero discrepancies — but self-classification standards differed between agents, which is why two of those figures are not comparable to each other.
Derived
Computed from other values
Structural counts from regex rules, ratios between recorded figures. The counting rule is stored with the number, because the rule is a judgement.
Not observable
The telemetry never existed
An absence of data, not a zero. Several experiments here have no token or cost figures at all, because the runtime exposed none to the running agent. They are labelled rather than omitted.
What this evidence base cannot support
- Any general claim about the industry. eleven experiments in one organisation, mostly by agents from one model family. That is not a sample of software engineering.
- Any percentage improvement you could put in a proposal. No experiment here was designed to establish an effect size, and the one that came closest has incomplete telemetry for its treatment condition.
- Portability beyond F#, TypeScript, C# and JavaScript. Nothing here touches any other ecosystem.
- Anything about teams. Every experiment was executed by a single agent or a single supervised session.
- A comparison against unconstrained development. No experiment here has a counterfactual arm in which the same requirements were built without the method.