Evidence
Controlled Experiment 3 — agent cost under hardening (corrected)
A paired A/B trial in which two isolated agents implemented the same frozen mission against the baseline and hardened architectures. The result that most complicates the programme's own thesis.
What this was
- Status
- Complete — this record supersedes an earlier uncorrected draft
- Repository
- kemiller2002/helix-note-application
- Branch
- experiment/agent-cost-comparison-v3
- Ending commit
- 8ac05fd25252
- Executed by
- Two freshly spawned, isolated subagents from the same model family, neither with access to the other's work or to the programme's prior reports. Orchestrated and independently re-verified by a third session.
Objective
Hold the mission constant and vary only the architecture, to see whether mechanical boundary hardening reduces what an AI agent actually spends to implement a change.
Hypothesis
Six preregistered hypotheses, from 'hardening reduces search effort' through 'mechanical feedback can substitute for agent inference'. Two were supported or partly supported; three were contradicted; one came out mixed.
Method
Condition A started from Experiment 1's baseline tag, Condition B from Experiment 2's hardened tag. Each was given the identical frozen mission — add imaging finding tracking, end to end — and executed by a separate isolated subagent. Both conditions were then re-verified by the orchestrating session against the actual committed diffs rather than accepted on self-report.
Recorded metrics
38 to 14
Search operations
Self-reported
- Measures
- Repository search operations performed by each agent while implementing the identical frozen mission, baseline architecture against hardened.
- Collected by
- Counted by each isolated subagent from its own execution log, then re-verified by the orchestrating session against committed diffs.
- Does not show
- One paired trial, one run per condition. Two agents with different investigation strategies; the reduction is confounded with that difference and is not attributable to architecture alone.
Sourcekemiller2002/state-directed-engineering / research/evidence/EV-HN-2026-0005--experiment-3-corrected-agent-cost.md — Corrected headline figures2 to 1
Repair loops
Self-reported
- Measures
- Cycles in which a fix had to be revised, baseline condition against hardened condition.
- Collected by
- Counted from each agent's own execution log and re-verified against committed diffs.
- Does not show
- A change from two to one in a single paired trial. This is a direction, not a rate, and the sample is one run per condition.
Sourcekemiller2002/state-directed-engineering / research/evidence/EV-HN-2026-0005--experiment-3-corrected-agent-cost.md — Corrected headline figures42.1% to 31.6%
As-experienced mechanical discovery rate
Self-reported
- Measures
- Each agent's own strict, temporally-ordered classification of which required sites it found mechanically rather than by reading, baseline against hardened.
- Collected by
- Self-classified by each agent during execution, under each agent's own strictness standard.
- Does not show
- The hardened condition scored LOWER here while spending less on search and repair. The two agents also applied different strictness standards, so these two figures are not directly comparable to each other — which is exactly why the experiment introduced a second, architecture-potential measure.
Sourcekemiller2002/state-directed-engineering / research/evidence/EV-HN-2026-0005--experiment-3-corrected-agent-cost.md — Corrected headline figures19 and 19
Required change sites
Instrumented
- Measures
- Normalised count of distinct locations the frozen mission required changing, in each condition.
- Collected by
- Counted from each condition's committed diff by the orchestrating session.
- Does not show
- Identical in both conditions. Hardening did not reduce the amount of work required; anything claiming otherwise contradicts this measurement.
Sourcekemiller2002/state-directed-engineering / research/evidence/EV-HN-2026-0005--experiment-3-corrected-agent-cost.md — Corrected headline figures4.0, third replication
Boundary files per semantic decision
Instrumented
- Measures
- Boundary Change Amplification, measured a third time — now across three experiments, three domain cases and two different architectures.
- Collected by
- Counted from committed diffs in each condition.
- Does not show
- Three replications in one codebase is a stable property of that codebase, not a law. No other application has been measured.
Sourcekemiller2002/state-directed-engineering / research/evidence/EV-HN-2026-0005--experiment-3-corrected-agent-cost.md — Corrected headline figures0
Discrepancies between agent self-report and independent re-verification
Instrumented
- Measures
- Build, test and architecture-check results each agent reported, checked against what their actual committed diffs produce.
- Collected by
- Every reported result was independently re-run by the orchestrating session against the committed code.
- Does not show
- This establishes that these agents reported their own results honestly. It says nothing about whether the work was correct — the same trial found a semantic no-op in both conditions.
Sourcekemiller2002/state-directed-engineering / research/evidence/EV-HN-2026-0005--experiment-3-corrected-agent-cost.md — Corrected headline figuresNot established
Agent token and wall-clock cost
Not observable
- Measures
- Whether the hardened architecture cost the agent less to work in.
- Collected by
- Harness-level telemetry was captured for both conditions, but the hardened condition was interrupted by a rate limit and resumed as a separate invocation, so its figures cover only the resumed portion — beginning after roughly 78% of its eventual log already existed.
- Does not show
- The raw figures imply 59% fewer tool uses, 68% fewer tokens and 59% less wall clock. They MUST NOT be read as a cost reduction. They are a lower bound on a partial run; the true total is unknown and could match or exceed the baseline. This question is open.
Sourcekemiller2002/state-directed-engineering / research/evidence/EV-HN-2026-0005--experiment-3-corrected-agent-cost.md — Harness-level telemetryabout 2.1x
Execution log step ratio
Instrumented
- Measures
- The ratio of numbered execution-log entries between the two conditions: 76 against 36.
- Collected by
- A direct recount of both logs, correcting an earlier draft of the same report that stated 167 entries for the baseline condition and implied a 4.6x ratio.
- Does not show
- Log entry counts are a proxy for effort shaped by each agent's own logging habits, not a measure of work done. The correction is published because the superseded figure circulated first.
Sourcekemiller2002/state-directed-engineering / research/evidence/EV-HN-2026-0005--experiment-3-corrected-agent-cost.md — Corrected log/step-count figuresFindings
- Search operations fell from 38 to 14 between the baseline and hardened conditions, a 63% reduction.
- Repair loops fell from 2 to 1.
- Boundary Change Amplification was 4.0 for a third independent time — across three separate experiments, three separate domain cases and, in this experiment, two separate architectures.
- Across the entire independent re-verification, every build, test and architecture-check result either agent reported was reproduced exactly against their committed diffs. Zero discrepancies were found between self-report and re-verification.
- Required change sites were identical in both conditions: 19 and 19.
What this refuted
- Higher mechanical discovery does not mean lower cost. The hardened condition had a LOWER as-experienced Mechanical Discovery Rate than the baseline — 31.6% against 42.1% — while simultaneously having substantially lower search and repair-loop costs. The two proxies moved in opposite directions in the same trial.
- Hardening did not reduce required work volume: required change sites were 19 in both conditions and Boundary Change Amplification was unchanged.
- Hardening did not close the semantic no-op class. The same present-but-inert implementation evaded every mechanical check on both architectures and was caught by integration tests only, in both conditions identically.
- The preregistered prediction that hardening would reduce silent structural omissions was contradicted: the outcome was identical in both conditions.
Anomalies, failures and confounders
- Condition B was interrupted mid-trial by an infrastructure rate limit and resumed as a separate invocation. Its recorded token, tool-use and wall-clock figures cover only the resumed portion, which began after roughly 78% of its eventual log already existed. Condition B's true total cost is unknown and could match or exceed Condition A's.
- Condition A spent roughly 19 to 21 of its 36.4 minutes on three client builds that Condition B never ran, for reasons unrelated to hardening.
- Condition A ran first in a shared container; Condition B ran second with warm caches and restore artifacts. A real, unquantified contributor to Condition B's shorter wall clock.
- The two agents applied different strictness standards to their own discovery self-classification, which is the primary reason their raw discovery rates are not directly comparable.
- An earlier draft of the source report stated a log entry count of 167 for Condition A. A direct recount gave 76. The step-count ratio is therefore about 2.1x, not the 4.6x the draft implied. The correction is recorded rather than quietly replaced.
Limitations
- One paired trial, one run per condition. A case study, not a statistically powered comparison. The source report makes no claim of statistical significance anywhere.
- Condition B's cost telemetry is a documented lower bound, not a total. The raw reductions it implies — 59% fewer tool uses, 68% fewer tokens, 59% less wall clock — must never be read as a confirmed cost reduction.
- Both conditions were executed by subagents from the same model family in the same environment. No cross-model or cross-tool comparison was made.
- Nothing here shows this architecture is superior to any other: no control implementation in a different architecture was built.
- Whether hardening reduces agent token or wall-clock cost remains OPEN, not answered.
Interpretation
This is the most useful experiment in the programme and the one that looks worst in a summary.
Set against its own predictions, it mostly failed. Of six preregistered hypotheses, three were
contradicted outright and one came out mixed. The hardened condition's headline discovery rate
went down. And the figure that would have been the programme's best marketing asset —
a 68% token reduction — cannot be claimed at all, because the run that produced it was
interrupted and resumed, and the counted portion began after most of the work was already done.
What makes it valuable is that it separated two things the programme had been treating as one.
Mechanical discovery rate had been serving as a proxy for cost. In this trial the hardened
condition had a lower discovery rate and lower search and repair costs simultaneously. A proxy
that moves opposite to the thing it proxies is not a proxy.
The confounders are worth reading in full, because they are the model for how this kind of
trial should be reported. Different investigation strategies between the two agents. Different
self-classification strictness, which the experiment noticed mid-flight and responded to by
introducing a second measure rather than by picking the flattering one. Nineteen minutes of one
condition's wall clock spent on builds the other never ran. Warm caches for whichever condition
went second. A shared database. Each of these is recorded as a reason to distrust the numbers,
by the people who produced the numbers.
There is one unambiguously good result buried in it: across the entire independent
re-verification, every build, test and architecture-check result either agent reported was
reproduced exactly against their committed diffs. Zero discrepancies. Both agents also
correctly reported their own token metrics as unavailable rather than estimating them. That
says something about whether agent self-reports can be trusted as evidence, and the answer here
was yes — which is only worth knowing because somebody checked.
And the semantic no-op appeared again, identically, in both conditions, caught by integration
tests only. Third trial, third time.