Evidence
Controlled Experiment 1 — boundary amplification on an unhardened architecture
How much of a new domain case's required propagation work a compiler can find on its own, and what happens to the part it cannot.
What this was
- Status
- Complete
- Repository
- kemiller2002/helix-note-application
- Branch
- experiment/observed-value-boundary-amplification
- Baseline tag
- state-programming-boundary-baseline-v1
- Baseline commit
- 182e0e22b071
- Ending commit
- a8d5a4fb81b8
- Executed by
- A human-supervised session acting as its own instrument. The isolated-subagent protocol did not exist yet; it was introduced in Experiment 3.
Objective
Add one new case to a closed domain alternative — VitalSignObserved, a new ObservedValue — on an architecture with no dedicated boundary-hardening mechanisms, and count every site that had to change, how each one was found, and whether anything could be silently missed.
Hypothesis
Compiler exhaustiveness checking finds the required propagation sites for a new closed-alternative case. Sites outside the type system's reach do not get found this way.
Method
A branch cut directly from the baseline tag. The mutation was implemented, every required handling site recorded as it was discovered, and each one classified by how it was found: a compiler error, or manual investigation. A deliberate probe then removed one fan-out arm and re-ran the build, the test suite and a direct database query to see whether anything would object.
Recorded metrics
8 of 12
Required sites found by the compiler alone
Self-reported
- Measures
- The share of required handling sites for a new closed-alternative case that a compiler exhaustiveness error pointed at directly — 66.7%.
- Collected by
- Each site was classified as it was discovered, as either a compiler error or manual investigation, during the implementation itself.
- Does not show
- One change class — a new case in a closed alternative — on one architecture, in one trial. It says nothing about change classes the type system does not model, and the session classified its own discoveries.
Sourcekemiller2002/state-directed-engineering / research/evidence/EV-HN-2026-0003--experiment-1-boundary-amplification.md — Key figures4.0
Boundary files per semantic decision
Instrumented
- Measures
- Boundary Change Amplification: the number of distinct boundary files that had to change for each single semantic decision.
- Collected by
- Counted from the committed diff of the experiment branch against its baseline tag.
- Does not show
- This is a measurement of one codebase's boundary structure, not a universal ratio. It is reported here because the same figure recurred in two later experiments, not because one measurement establishes a constant.
Sourcekemiller2002/state-directed-engineering / research/evidence/EV-HN-2026-0003--experiment-1-boundary-amplification.md — Key figures0
Repair loops
Self-reported
- Measures
- Cycles in which a fix had to be revised because it failed to compile or failed a test.
- Collected by
- Counted during implementation; every fix compiled and passed on its first attempt.
- Does not show
- Zero repair loops means the mechanically-discovered fixes were mechanical. It does not mean the change was correct: the same experiment's own probe showed a missing boundary arm passing every check.
Sourcekemiller2002/state-directed-engineering / research/evidence/EV-HN-2026-0003--experiment-1-boundary-amplification.md — Key figures0 of 3
Checks that objected to a deliberately removed boundary arm
Instrumented
- Measures
- Of the compiler, the existing test suite and the running application, how many produced any signal when a required fan-out arm was removed.
- Collected by
- A deliberate probe: remove the arm, build, run the tests, query the database directly, then restore.
- Does not show
- One probe, one arm, one architecture. It establishes that this failure is real and reproducible here, not how often it occurs in general.
Sourcekemiller2002/state-directed-engineering / research/evidence/EV-HN-2026-0003--experiment-1-boundary-amplification.md — Key figuresNOT OBSERVABLE
Tool calls and tokens
Not observable
- Measures
- Agent tool-call and token counts for this experiment.
- Collected by
- Not collected. The executing session had no structured telemetry access, and the source report labels its work-metrics table a manual reconstruction rather than a counter reading.
- Does not show
- This is an absence of data, not a zero. No cost comparison involving this experiment is possible, and none is made.
Sourcekemiller2002/state-directed-engineering / research/evidence/EV-HN-2026-0003--experiment-1-boundary-amplification.md — Key figuresFindings
- Eight of twelve required handling sites — 66.7% — were found by compiler exhaustiveness checking (FS0025) alone, on an architecture with no boundary hardening of any kind.
- The four sites the compiler could not find were reached by SQL CHECK-constraint archaeology, by inferring symmetry with an already-fixed sibling function, and by reviewing test coverage. None of these is a mechanism; all three are a person looking.
- The one fact the compiler protects — the shape of the closed alternative itself — had zero duplication. The two facts outside its reach were duplicated four times and twice respectively, with no automated agreement check between the copies.
- Boundary Change Amplification was 4.0 boundary files per semantic decision, and 7.0 mechanical boundary edits per semantic decision.
- Every fix compiled and passed on the first attempt: zero repair loops.
- The semantic architecture check passed both before and after, and kept passing through the deliberate-omission probe.
What this refuted
- The silent-omission probe refuted the idea that this architecture's existing checks would catch a missing boundary arm. Removing the fan-out arm produced zero compiler errors, zero test failures and zero runtime exceptions. The candidate was durably recorded as accepted while the clinical fact it represented was never written anywhere retrievable.
Anomalies, failures and confounders
- The completion tag could not be pushed to the remote: HTTP 403 on the tag ref. It exists locally on the recorded commit only, so the experiment is reproducible from the branch commit rather than from a tag listing on a fresh clone.
- Tool-call and token counts are recorded as NOT OBSERVABLE. The executing session had no structured telemetry access, and the work-metrics table in the source report is an explicitly-labelled manual reconstruction rather than a counter reading.
Limitations
- A single trial, a single new case, on a single architecture. The source report itself declines to claim that one data point establishes a distribution.
- The executing session was its own instrument: it recorded its own discovery classifications. Experiments 2 and 3 inherited this limitation; only Experiment 3 added independent re-verification against committed diffs.
- 66.7% is a mechanical discovery rate for one class of change — a new case in a closed alternative. It says nothing about change classes the type system does not model.
Interpretation
The headline here is often read as "the compiler found two thirds of it". The more useful
reading is the other third.
Four sites required a person to go looking, and the record of how they were found is
the interesting part: database constraint archaeology, noticing a symmetry with a sibling
function that had already been fixed, and reviewing test coverage. None of those is a
mechanism. All three are a competent person being thorough, and none of them would have
survived a tired afternoon.
The distribution explains itself. The one fact under the compiler's protection had zero
duplicated copies. The two facts outside it had four and two. That is not a coincidence about
this codebase — it is what happens to a fact that nothing forces to agree with itself.
The silent-omission probe is what turned this from a measurement into a research programme.
Everything passed. The build, the tests, the runtime, the architecture check. A clinical fact
was accepted into the system and written nowhere retrievable, and the only thing that noticed
was an experiment deliberately looking for it.
This experiment set both of the directions everything after it went: mechanise what the type
system can reach (Experiment 2), and find out what still gets through when you do
(Experiment 2's Phase 20, and every trial since).