Concept 9

Evidence and traceability

A result that cannot be checked is an assertion. Ordo treats the recording of evidence as part of the engineering, not as documentation produced afterwards.

Six things every published number needs

Definition

What exactly does it measure?

Measurement

How was it collected, by what instrument?

Scope

Which experiment, branch and commit does it apply to?

Value

What was actually observed, unrounded and unreformatted?

Provenance

Where is the artifact a reader can open?

Limitation

What must a reader not conclude from it?

The last one is the one that gets dropped, and it is the one that makes the other five safe to publish. On this site it is not optional in the ordinary sense: a metric with no stated limitation cannot be represented in the evidence manifest, and the build fails.

Missing is not zero

The rule Ordo inherited from its own experimental record: a metric whose telemetry genuinely does not exist is labelled NOT OBSERVABLE and stays visible. It is never estimated, and never quietly dropped from the table.

NOT OBSERVABLE

Token and cost totals

Not observable

Measures
Token counts, cost and context occupancy for this run.
Collected by
Not collected: this runtime exposes none of them to a running session. Recorded as unavailable at every checkpoint, and again explicitly at experiment close.
Does not show
An absence of data, not a zero. No cost comparison with any other experiment on this site is possible.
Sourcekemiller2002/SDE-Engineering-Trial-2-Greenfield-Construction / experiment/TELEMETRY.md at claude/greenfield-engineering-experiment-2-ee44qz — T6 — Experiment Closedprivate repository

NOT OBSERVABLE

Tool calls and tokens

Not observable

Measures
Agent tool-call and token counts for this experiment.
Collected by
Not collected. The executing session had no structured telemetry access, and the source report labels its work-metrics table a manual reconstruction rather than a counter reading.
Does not show
This is an absence of data, not a zero. No cost comparison involving this experiment is possible, and none is made.
Sourcekemiller2002/state-directed-engineering / research/evidence/EV-HN-2026-0003--experiment-1-boundary-amplification.md — Key figures

Not established

Agent token and wall-clock cost

Not observable

Measures
Whether the hardened architecture cost the agent less to work in.
Collected by
Harness-level telemetry was captured for both conditions, but the hardened condition was interrupted by a rate limit and resumed as a separate invocation, so its figures cover only the resumed portion — beginning after roughly 78% of its eventual log already existed.
Does not show
The raw figures imply 59% fewer tool uses, 68% fewer tokens and 59% less wall clock. They MUST NOT be read as a cost reduction. They are a lower bound on a partial run; the true total is unknown and could match or exceed the baseline. This question is open.
Sourcekemiller2002/state-directed-engineering / research/evidence/EV-HN-2026-0005--experiment-3-corrected-agent-cost.md — Harness-level telemetry

The third one is the hard case. That experiment did produce token and wall-clock figures. Reported raw, they imply a 68% token reduction, which is by some distance the most quotable number this programme has generated. It is published here as not established, because the condition that produced it was interrupted by a rate limit and resumed, and the counted portion began after roughly 78% of its eventual work already existed.

The true total is unknown and could exceed the baseline. The source record says so in bold, twice, and calls the caveat load-bearing and never to be dropped in any downstream summary. This is that downstream summary.

Contradictions stay in the record

When a measurement refutes something the programme believed, the refuted proposition is not deleted. It stays in the evidence map permanently, classified Contradicted, cross-linked to the record that refuted it, so that a future agent cannot silently re-propose it without seeing why it was rejected.

Five propositions on this site carry that status. They are on the research page with everything else, not in an appendix.

Corrections are published, not applied quietly

An earlier draft of one experiment's report stated a step count of 167 where a direct recount gave 76 — a 4.6x ratio where the real figure is about 2.1xExecution log step ratio. The correction is recorded as a correction, with both numbers, and a decision record names the corrected version as authoritative.

Similarly, a metrics dictionary's first version claimed agent cost data was unavailable to a running session. It was wrong; the API exposes it. The correction is in that dictionary's changelog rather than edited over the original, because silent redefinition makes before-and-after comparisons meaningless.

What an agent should do differently

How this is most often got wrong

The tidy summary. Several findings collapsed into one confident sentence, with the confounders left in a document nobody reads. Every experiment on this site has a limitations section, and they are the most useful part of each page.

Reporting only what was measurable. An experiment that publishes its available metrics and omits the rest has published a filtered result without saying so.