Evidence
C# state-system trial — the same domain, structured explicitly
The full time-entry domain built in C# against an explicit state model, reaching a first clean pass with no domain defects and no rework.
What this was
- Status
- Complete
- Repository
- kemiller2002/time-entry-state-machine
- Branch
- main
- Ending commit
- 73a63f1ac319
- Executed by
- Claude Code (Sonnet 5), single continuous session, .NET SDK 10, C# latest.
Objective
Recover the complete current requirement set, model it explicitly as states and legal transitions, then implement it in C# — and record what the compiler could and could not find along the way.
Hypothesis
Explicitly modelling state before implementing concentrates the work in semantic decisions and lets mechanical checks find the propagation.
Method
Requirements were recovered from the repository into a 398-line specification, modelled as a state machine with a 14-row legal-transition table, and only then implemented. Every phase boundary was timestamped, every discovery classified as compiler-found or search-found, and an adversarial state-system test run against the result.
Recorded metrics
all of them
Ripple sites found by the compiler rather than by searching
Self-reported
- Measures
- Locations needing change as a consequence of another change, and whether each was reported by the compiler or found by a manual search.
- Collected by
- Classified per location during the trial. Every ripple location was compiler-discovered; the manual-search count was zero.
- Does not show
- One trial with one ripple-producing change. It shows the compiler covered this trial's ripple, not that it covers ripple in general — the controlled baseline trial found arithmetic changes that no type system catches.
Sourcekemiller2002/time-entry-state-machine / effort-experiment/c-sharp-state/EXPERIMENT-REPORT.md at 73a63f1ac319 — I. Compiler Ripple Analysisprivate repository0
Runtime domain defects
Self-reported
- Measures
- Defects in domain logic reaching a run, as distinct from test-authoring or language-tooling errors.
- Collected by
- Counted across the whole trial, with all three repair loops classified and none of them a domain-logic error.
- Does not show
- A single uninterrupted first pass with no later requirement change to force rework. The trial's own report declines to read this as evidence that the architecture reduces rework.
Sourcekemiller2002/time-entry-state-machine / effort-experiment/c-sharp-state/EXPERIMENT-REPORT.md at 73a63f1ac319 — G. Metricsprivate repository0%
Rework ratio
Derived
- Measures
- Proportion of completed work replaced because earlier work turned out to be wrong.
- Collected by
- Derived from the per-file classification: no file was rewritten, and no semantic decision was revised.
- Does not show
- No later requirement arrived to force rework during this trial, so this is the absence of a cause, not evidence of resistance to one.
Sourcekemiller2002/time-entry-state-machine / effort-experiment/c-sharp-state/EXPERIMENT-REPORT.md at 73a63f1ac319 — G. Metricsprivate repository10 of 10
Cold-start questions answerable without opening source
Self-reported
- Measures
- Whether a fresh engineer could answer what the application does, where authoritative state lives, which transitions are legal, where each workflow is implemented and how it is tested — from the generated documents alone.
- Collected by
- Evaluated question by question against the requirement specification, state model, execution map and state-system test report, holding source code in reserve.
- Does not show
- Evaluated after a documentation gap found by the trial's own adversarial test had been closed, so it reflects the corrected state rather than the first pass. It was also self-assessed, not tested on an actual fresh reader.
Sourcekemiller2002/time-entry-state-machine / effort-experiment/c-sharp-state/EXPERIMENT-REPORT.md at 73a63f1ac319 — M. Cold-Start Reconstructionprivate repository18
Repository searches across the whole trial
Instrumented
- Measures
- Find and grep operations performed during requirement recovery, modelling, implementation and verification combined.
- Collected by
- Recounted from the session transcript, superseding an earlier lower estimate.
- Does not show
- Not comparable to the conventional trials, which did not record search counts the same way. It is a figure about this trial, not a measured reduction.
Sourcekemiller2002/time-entry-state-machine / effort-experiment/c-sharp-state/EXPERIMENT-REPORT.md at 73a63f1ac319 — G. Metricsprivate repository0
Context expansions beyond the planned reading list
Self-reported
- Measures
- Occasions where in-hand context proved insufficient and the agent had to widen its search to proceed.
- Collected by
- Counted during the trial.
- Does not show
- One agent working from a specification it had itself just produced, which is a favourable condition. It does not measure what a fresh agent would need.
Sourcekemiller2002/time-entry-state-machine / effort-experiment/c-sharp-state/EXPERIMENT-REPORT.md at 73a63f1ac319 — G. Metricsprivate repositoryFindings
- Zero runtime domain defects and a rework ratio of 0%: no completed work was replaced because earlier work was wrong.
- Every ripple location was found by the compiler. A genuine naming ambiguity surfaced at both affected call sites in a single build, with exact file and line, and none of the trial's ripple locations required a manual search.
- All sixteen domain files touched contained semantic decisions; none contained only mechanical propagation.
- All ten cold-start questions — what the application does, where authoritative state lives, which transitions are legal, where each workflow is implemented, how it is tested — were answerable from the four generated documents without opening a single source file.
- Sixty-five tests, five build attempts, three repair loops, none of them a domain-logic error.
- Eighteen search operations across the whole trial, with zero context expansions beyond the planned reading list.
What this refuted
- The clean first pass is not evidence that the architecture reduces rework. The trial's own report states this plainly: with no later requirement change to force one, Semantic Change Coverage is NOT APPLICABLE rather than zero, and a fair claim would need a follow-up change-task trial that this experiment did not run.
Anomalies, failures and confounders
- Two of the three repair loops were C#-specific language and tooling constraints unrelated to domain modelling. The third was a test that wrongly assumed record equality deep-compares a collection member — a gap no compiler warning exists for in any configuration, caught only by running the test.
- The agent escalated once, to a human, over a governing document that was missing from the repository.
- Timing figures come from phase-boundary timestamps the agent chose to record, not from independent instrumentation. The report attaches that caveat rather than overstating precision.
- Token, cost and total-token figures are NOT AVAILABLE — not exposed to the agent.
Limitations
- A single uninterrupted implementation pass by one agent. No later requirement change forced rework, which is precisely where the two conventional trials produced their most informative findings.
- Comparing this trial against the conventional rebuilds is a comparison of persona as much as of architecture: this trial deliberately reached for the strongest available C# patterns, and those trials deliberately reached for ordinary ones.
- Not evidence about C# versus F#: no F# implementation of this same state-guided design existed when the trial ran.
- Adversarial testing found what it found in the time available. Absence of a further finding is not proof there is none.
Interpretation
The headline numbers are clean — no runtime domain defects, nothing rewritten, every ripple site
reported by the compiler — and the report's own limitations section is the reason to trust them
rather than the reason to discount them.
What this trial demonstrates is that requirements-first construction is achievable and produces a
navigable result. Forty-six minutes recovering requirements into a 398-line specification, sixteen
minutes modelling states and a fourteen-row legal-transition table, and only then an hour
implementing. The structural consequence is visible in the metrics: all sixteen domain files
touched contained semantic decisions and none contained only mechanical propagation, because the
decisions had already been made before any code was written.
The cold-start result is the one with the clearest practical value. Ten out of ten questions a
fresh engineer would need answered — what the application does, where authoritative state lives,
which transitions are legal, where each workflow is implemented, how it is tested — were
answerable from the four generated documents without opening a source file. The report is
scrupulous that this reflects the documentation state after a gap its own adversarial
test had found was closed, and that the assessment was made by the agent rather than by an actual
fresh reader.
What this trial explicitly does not demonstrate is the thing it would be most
commercially useful to claim. There was no later requirement change, so there was no ripple to
measure, so Semantic Change Coverage is recorded as not applicable rather than as zero.
The report says outright that a fair claim about reduced rework needs a follow-up change-task
trial against this codebase, and that inferring it from a clean first pass is not the same thing.
We have not run that trial, and we are not making that claim.
One detail is worth keeping for its own sake. The trial's third repair loop was a test that
assumed C#'s record equality deep-compares a collection member. It does not. No compiler warning
exists for that in any configuration — the only way to find it was to run the test and watch it
fail. A methodology that leans on mechanical checking needs that example in its own record.