Evidence
Conventional C# rebuild — ordinary practice, requirement by requirement
The same domain rebuilt the way a great deal of production C# is actually written, with every mistake recorded live and classified by what caught it.
What this was
- Status
- Complete
- Repository
- kemiller2002/time-entry-state-machine
- Branch
- main
- Ending commit
- 73a63f1ac319
- Executed by
- A single session building under a stated persona and a no-look-ahead rule.
Objective
Find out what kind of mistake ordinary C# practice allows to exist, and which mechanism has to catch it.
Hypothesis
An explicit-state domain converts classes of mistake that conventional code can only catch by disciplined manual review into mistakes a compiler catches automatically.
Method
Built under an explicit persona — a competent professional C# developer who has never worked in a functional language, reaching for mutable classes and bare primitives by default. Deliberately not a strawman. Each requirement was implemented in the order the specification lists it, with no reading ahead and nothing generalised in anticipation. Whatever rework a later requirement forced was the data point.
Recorded metrics
6
Mistakes in the conventional C# rebuild
Self-reported
- Measures
- Rework-driving mistakes across a 1,645-line rebuild, built requirement by requirement with no reading ahead.
- Collected by
- Recorded live per requirement, with build attempts, test runs to green, rework and the mechanism that caught each mistake.
- Does not show
- The raw count matching the F# domain's is the least interesting thing about this number. What differs is which mechanism caught each one.
Sourcekemiller2002/time-entry-state-machine / effort-experiment/EFFORT-LOG.md — Totals — the complete hodgepodge buildprivate repository3 of 6
C# mistakes caught by the compiler
Self-reported
- Measures
- How many of the conventional C# rebuild's six rework-driving mistakes the compiler caught by itself: a csproj ImplicitUsings error, a var-hidden signature change, and F7's CS1628.
- Collected by
- Classified per mistake as it was found, then re-derived from the log's own enumeration of the six. CORRECTED from '4 of 6'. The source log's tally of 4 includes F1's eight compiler-reported rename sites, which are the compiler propagating an INTENDED change rather than catching a mistake, and which the log's own F1 row records as '0 shipped'. That figure is published separately as M-TE-CS-RIPPLE, where it belongs.
- Does not show
- Two caveats a reader should have. First, the mechanism tallies do not partition cleanly: F7 was caught twice, by a pre-build read-through and then by the compiler, so seven catch-events cover six mistakes. Second, and more important: every one of the sharpest bugs — swapped string arguments, a stale revision in a loop, a signature change hidden behind var — was outside what the compiler could catch even in principle, and was found only by a person re-reading a diff.
Sourcekemiller2002/time-entry-state-machine / effort-experiment/EFFORT-LOG.md — Totals — the complete hodgepodge buildprivate repository~14
Call sites where a rename silently changed meaning
Self-reported
- Measures
- Sites in the conventional C# rebuild where duration was passed as a bare positional integer, so the same source text meant something six times different after the unit change.
- Collected by
- Counted by hand during the change, because no mechanism surfaced them. The compiler caught eight direct member accesses and none of these.
- Does not show
- An approximate count of sites requiring manual re-interpretation, not a defect count. Every one was correctly migrated in this trial — the finding is that nothing mechanical distinguished a correct migration from a wrong one, not that wrong ones shipped.
Sourcekemiller2002/time-entry-state-machine / effort-experiment/EFFORT-LOG.md at 73a63f1ac319 — F1 — exact-minute recording + derived billingprivate repository2
Values that survived the change correct by coincidence
Self-reported
- Measures
- Call sites left unchanged not because they were verified but because their literal happened to remain valid under the new unit.
- Collected by
- Identified by reading each site's intent during the manual migration pass.
- Does not show
- Two sites in one change in one codebase. It illustrates the failure mode rather than measuring its frequency.
Sourcekemiller2002/time-entry-state-machine / effort-experiment/EFFORT-LOG.md at 73a63f1ac319 — F1 — exact-minute recording + derived billingprivate repository1 of 6
C# mistakes caught by a test run
Self-reported
- Measures
- Mistakes in the conventional C# rebuild first surfaced by running the test suite rather than by the compiler or a person.
- Collected by
- Classified per mistake as it was found. The one case was C1's test arithmetic.
- Does not show
- A single case. It is published so the mechanism columns add up in public rather than only in the log.
Sourcekemiller2002/time-entry-state-machine / effort-experiment/EFFORT-LOG.md — Totals — the complete hodgepodge buildprivate repository3 of 6
C# mistakes caught only by a person re-reading the diff
Self-reported
- Measures
- Mistakes in the conventional C# rebuild caught by a deliberate pre-build read-through, with no mechanism involved.
- Collected by
- Classified per mistake as it was found: a missing label-active check, F2's argument-order swap, and F7's stale revision in a loop.
- Does not show
- This is the figure the whole comparison turns on. The read-through discipline held for the entire build by design, and a developer under deadline pressure — the persona this trial set out to simulate — is the least likely to apply it consistently. F7 also appears in the compiler count, having been caught twice.
Sourcekemiller2002/time-entry-state-machine / effort-experiment/EFFORT-LOG.md — Totals — the complete hodgepodge buildprivate repository8 of ~22
Call sites the C# compiler reported on the unit change
Self-reported
- Measures
- Sites the compiler named when duration changed from six-minute units to exact minutes, out of roughly twenty-two that needed attention.
- Collected by
- Counted during the change. Eight were direct member accesses the compiler reported; the rest were bare positional integers it could not see.
- Does not show
- This is the compiler propagating an intended change, not catching a mistake — a distinction the source log's mistake tally blurred and this record separates. Its complement is M-TE-CS-SILENT.
Sourcekemiller2002/time-entry-state-machine / effort-experiment/EFFORT-LOG.md — F1 — exact-minute recording + derived billingprivate repository1,645 lines
Conventional C# rebuild size
Self-reported
- Measures
- Final size of the conventional C# rebuild across five files, with 59 passing tests.
- Collected by
- Counted at the end of the build.
- Does not show
- Published so the mistake counts can be read against comparable volume rather than in the abstract.
Sourcekemiller2002/time-entry-state-machine / effort-experiment/EFFORT-LOG.md — Totals — the complete hodgepodge buildprivate repositoryFindings
- The unit-of-measure change is the finding. Renaming a duration field caught eight call sites in the compiler — and left roughly fourteen more where duration was passed as a bare positional integer, so identical source text silently changed meaning by a factor of six.
- Those fourteen sites had to be found by hand and each one's intent re-read to decide what it had meant. Two were left unchanged not because they were verified but because they happened to remain valid by coincidence.
- The build was clean and all thirty tests passed while that ambiguity was in flight. Nothing mechanical distinguished a correctly-migrated call site from a silently wrong one.
- Of six rework-driving mistakes, the compiler caught four. Every one of the sharpest — two swapped same-typed string arguments, a stale revision captured in a loop, a signature change hidden behind an inferred local — was structurally beyond what a compiler can catch, and was found only by a person re-reading a diff before running it.
- Building the correction path surfaced a real error in the reconstructed requirements themselves, which were corrected rather than left wrong.
What this refuted
- On the tracked metric this requirement looks better here than in the explicitly-typed codebase: one build attempt, one test run, zero mistakes that survived. The log records that headline as true but incomplete, and is explicit that the metric alone is misleading for a rename-shaped change under a primitive-typed design.
Anomalies, failures and confounders
- The trial's own tracking table under-counted the requirement list twice, causing two requirements to be built out of order — a property of the bookkeeping, not the code.
- The re-reading discipline that caught the sharpest bugs held for the entire build by design. The persona being simulated — a competent professional under deadline — is the group least likely to apply it consistently.
Limitations
- One person, one session, one domain, style varied under a stated persona rather than by recruiting developers who work that way.
- Written by someone who already knew the answer. This is a named threat to validity in the record, and the reason the persona and no-look-ahead rules exist.
- Mistake counts are not comparable across trials without accounting for order: each rebuild benefited from what the previous one learned.
- Nothing here measures cost, schedule, or defect escape in production.
Interpretation
On the metric this experiment tracked, the conventional build handled the hardest change better
than the explicitly-typed codebase did: one build attempt, one test run, nothing that failed.
The log records that headline and then spends the rest of the entry explaining why it is the
wrong number to look at.
The change renamed a duration field and altered what its unit meant. The compiler found eight
sites — every place the old member name appeared. It could not find, and did not find, the
roughly fourteen places where duration arrived as a bare positional integer. There, the literal
10 had meant ten six-minute units and now meant ten minutes: a sixfold change in
real duration, expressed in source text that did not alter by a single character.
Each of those had to be found by hand and then interpreted — was this 10 standing in for
an hour, or was it already a small literal that means the same either way? Most were migrated
correctly. Two were left untouched, and the log is careful about why: not because they were
checked and found correct, but because they happened to remain valid in the new scheme by
coincidence. That is the most instructive sentence in the trial.
Throughout, the build was clean and thirty tests passed. Nothing mechanical distinguished a
correctly migrated call site from a silently wrong one. A developer trusting a green build as
evidence of correctness — which is the ordinary and reasonable thing to do — would have had no
signal at all.
The wider pattern across the whole rebuild says the same thing more generally. The compiler
caught four of six rework-driving mistakes, and every one of the sharpest was a category it
cannot reach in principle: two swapped string arguments of the same type, a stale revision
captured in a loop, a signature change hidden behind an inferred local. All three were caught by
a person deliberately re-reading a diff before running it. That discipline held perfectly here,
by design — and the persona being simulated, a competent professional under deadline, is exactly
the group least likely to apply it every time.