Evidence
Greenfield Trial 1 — instrumented construction from frozen requirements
A time-entry system built from a frozen requirements package, with development-metrics instrumentation established before any implementation code was written.
What this was
- Status
- Complete
- Repository
- kemiller2002/SDE-Engineering-Trial-1-Greenfield-Construction
- Branch
- claude/dev-metrics-instrumentation-dfnzhq
- Ending commit
- 77a6952a5b4f
- Executed by
- Claude Code, configured model claude-sonnet-5, effort level high, permission mode auto, single remote session.
Objective
Take a bounded application from a frozen requirements package through to verified software, while capturing the raw evidence needed to compare architectural treatments later — rather than reconstructing numbers from memory afterwards.
Hypothesis
Instrumentation established before implementation produces comparable evidence. Instrumentation added afterwards does not.
Method
A metrics dictionary defining every metric's unit, collection mechanism, raw evidence source, derivation formula and known limitations was written and frozen first. Builds and test runs were then run through wrapper scripts that record exit code and duration to immutable JSON/log pairs; phase transitions, defects and AI reasoning errors were logged as they happened; repository snapshots were taken at each checkpoint. Only then was the application implemented.
Recorded metrics
59 / 59
Acceptance tests passing
Instrumented
- Measures
- Tests passing in the final recorded run, with zero failed, zero skipped and zero cancelled.
- Collected by
- Captured by a test wrapper script to an immutable JSON and log pair recording the command, timestamps, duration and exit code.
- Does not show
- A passing suite says what was verified, not what was missed. This suite was written by the same agent that wrote the implementation, and shares its blind spots.
Sourcekemiller2002/SDE-Engineering-Trial-1-Greenfield-Construction / metrics/raw/tests/2026-09-02T23-44-05-163Z_final-suite_.json at claude/dev-metrics-instrumentation-dfnzhqprivate repository$19.04
Recorded agent cost
Instrumented
- Measures
- Platform-reported cumulative cost for the session, in US dollars — the platform's own figure, not a token-math estimate.
- Collected by
- Read from the session usage API and written to the agent-session log with the reading's timestamp and source recorded alongside it.
- Does not show
- A cumulative mid-session reading covering BOTH the metrics-instrumentation task and the application implementation, taken while the session was still running. It cannot be attributed to the implementation alone and is not a total.
Sourcekemiller2002/SDE-Engineering-Trial-1-Greenfield-Construction / metrics/raw/agent-sessions.jsonl at claude/dev-metrics-instrumentation-dfnzhqprivate repository60,303,576
Recorded tokens
Instrumented
- Measures
- Total tokens for the session: 1,503 fresh input, 227,066 output, 58,825,031 cache reads and 1,249,976 cache writes.
- Collected by
- Read from the session usage API at the same checkpoint as the cost figure.
- Does not show
- Overwhelmingly cache reads, which are not comparable to fresh input tokens in cost or in meaning. Like the cost figure, it covers two tasks and is a mid-session reading.
Sourcekemiller2002/SDE-Engineering-Trial-1-Greenfield-Construction / metrics/raw/agent-sessions.jsonl at claude/dev-metrics-instrumentation-dfnzhqprivate repository4
Distinct legal transitions
Derived
- Measures
- Distinct from-state to to-state pairs the entry lifecycle permits: create, void, restore, and supersede by split or merge.
- Collected by
- Counted by a structural snapshot script whose counting rule is stored in the snapshot record itself.
- Does not show
- A regex-derived count over one implementation, with the counting rule recorded because it is a judgement. No AST parser existed when the metric was defined.
Sourcekemiller2002/SDE-Engineering-Trial-1-Greenfield-Construction / metrics/raw/state-system/2026-09-03T00-59-15-753Z_fd2a013cb32a.json at claude/dev-metrics-instrumentation-dfnzhqprivate repository24
Distinguishable rejection states
Derived
- Measures
- Named diagnostic variants: distinct reasons the system can refuse an action, each one a value rather than a thrown exception or a null.
- Collected by
- Counted by the same structural snapshot script, over the diagnostic union's variants.
- Does not show
- Counts how many refusals are distinguishable, not whether each one is correct or whether the set is complete.
Sourcekemiller2002/SDE-Engineering-Trial-1-Greenfield-Construction / metrics/raw/state-system/2026-09-03T00-59-15-753Z_fd2a013cb32a.json at claude/dev-metrics-instrumentation-dfnzhqprivate repository5
Agent reasoning errors logged
Self-reported
- Measures
- Misread requirements or invented rules, logged separately from ordinary defects whether or not they produced observable misbehaviour.
- Collected by
- Logged at the moment each was found, with category, what detected it, and the correction.
- Does not show
- A count of errors the agent noticed and recorded about itself. Errors it did not notice are, by construction, absent from this number.
Sourcekemiller2002/SDE-Engineering-Trial-1-Greenfield-Construction / metrics/ai-errors/AI-ERROR-LOG.jsonl at claude/dev-metrics-instrumentation-dfnzhqprivate repositoryFindings
- All 59 acceptance tests passed on the final run, with the type checker clean, both captured as immutable instrument records rather than as a claim in prose.
- The time-entry domain was built with three entry states, four distinct legal transitions, seven guards, seven entry capabilities and twenty-four distinguishable diagnostic variants.
- One illegal transition is explicitly represented and directly tested rather than merely absent from the code: nothing leaves the Superseded state, and two acceptance criteria assert the refusal.
- Five AI reasoning errors were logged separately from ordinary defects. Four were caught by the compiler, by inspection or by an integration test; one — designing a commit hook to append to a tracked file, creating an unbreakable chase where each commit needs another commit to record it — was caught only by the user's own tooling, not by self-review.
- Real, platform-reported agent cost was captured for this session, which earlier work had wrongly assumed was unavailable to a running agent.
What this refuted
- The metrics dictionary's own first version claimed per-session token and cost data was not programmatically available. That was wrong, and the correction is recorded in the metrics changelog rather than edited away: the platform's session API does expose cumulative usage and actual cost. What it does not expose is a per-turn breakdown, tool-call counts, retries or API errors, which remain best-effort.
Anomalies, failures and confounders
- The recorded cost reading covers both the metrics-instrumentation task and the application implementation in one figure, and was taken mid-session while the session was still running. It is a cumulative reading at a point in time, not a total for the implementation alone.
- Tool-call counts, retries, API errors and context resets are recorded as null rather than estimated: the session API does not expose them and no transcript was available.
- No AST-aware parser existed for counting functions, types or complexity at the time the dictionary was written, because the implementation language had not been chosen. The structural counts recorded are regex-derived and the counting rule is stored alongside each number.
Limitations
- This trial has no comparison arm. It measures one construction, thoroughly; it does not compare that construction against an alternative, so nothing here establishes that the method caused any of the outcomes.
- Test counts and pass rates say what was verified, not what was not. A suite written by the same agent that wrote the implementation shares that agent's blind spots — a limitation Greenfield 2B later measured directly.
- Cost and token figures are a cumulative mid-session reading covering two different tasks and cannot be attributed to the implementation alone.
- Several metric families defined in the dictionary were never populated, because the instruments for them did not exist. The dictionary records the gaps rather than dropping the rows.
Interpretation
This trial's contribution is not its output. It is that the instruments were built and frozen
before the work started.
A metrics dictionary defining every metric's unit, collection mechanism, raw evidence source,
derivation formula and known limitations was written first, with a rule that no definition
changes without a changelog entry. Builds and tests then ran through wrappers that write
immutable evidence pairs. The result is that the numbers here are readings rather than
recollections — which is the difference between this trial and the first two, whose work
metrics are explicitly labelled manual reconstructions.
The dictionary was also wrong about something, and that is worth more than most of what it got
right. Its first version stated that per-session cost data was not available to a running
agent. It is; the session API exposes it. The correction sits in the changelog next to the
original claim rather than replacing it, because a dictionary that silently redefines its own
metrics makes every before-and-after comparison meaningless.
The agent-error log is the other thing worth copying. Five reasoning errors recorded separately
from defects — a misread requirement or an invented rule, whether or not it produced
misbehaviour. One of them is a small masterpiece of the genre: a commit hook designed to append
each commit's evidence to a tracked file, which requires another commit to record, forever. The
agent's own self-review did not catch it. The user's tooling did.
The cost figure needs its caveat read. It covers two different tasks in one number, taken
mid-session. It is published because it is real and rare; it is not a cost-per-feature figure
and nothing on this site treats it as one.