Concept 9
Evidence and traceability
A result that cannot be checked is an assertion. Ordo treats the recording of evidence as
part of the engineering, not as documentation produced afterwards.
Six things every published number needs
Definition
What exactly does it measure?
Measurement
How was it collected, by what instrument?
Scope
Which experiment, branch and commit does it apply to?
Value
What was actually observed, unrounded and unreformatted?
Provenance
Where is the artifact a reader can open?
Limitation
What must a reader not conclude from it?
The last one is the one that gets dropped, and it is the one that makes the other five safe
to publish. On this site it is not optional in the ordinary sense: a metric with no stated
limitation cannot be represented in the evidence manifest, and the build fails.
Missing is not zero
The rule Ordo inherited from its own experimental record: a metric whose telemetry genuinely
does not exist is labelled NOT OBSERVABLE and stays visible. It is never
estimated, and never quietly dropped from the table.
The third one is the hard case. That experiment did produce token and wall-clock figures.
Reported raw, they imply a 68% token reduction, which is by some distance the most quotable
number this programme has generated. It is published here as not established,
because the condition that produced it was interrupted by a rate limit and resumed, and the
counted portion began after roughly 78% of its eventual work already existed.
The true total is unknown and could exceed the baseline. The source record says so in bold,
twice, and calls the caveat load-bearing and never to be dropped in any downstream summary.
This is that downstream summary.
Contradictions stay in the record
When a measurement refutes something the programme believed, the refuted proposition is not
deleted. It stays in the evidence map permanently, classified Contradicted, cross-linked to
the record that refuted it, so that a future agent cannot silently re-propose it without
seeing why it was rejected.
Five propositions on this site carry that status. They are on the
research page with everything else, not in an appendix.
Corrections are published, not applied quietly
An earlier draft of one experiment's report stated a step count of 167 where a direct recount
gave 76 — a 4.6x ratio where the real figure is
about 2.1xExecution log step ratio. The correction is recorded as a correction, with both numbers, and a
decision record names the corrected version as authoritative.
Similarly, a metrics dictionary's first version claimed agent cost data was unavailable to a
running session. It was wrong; the API exposes it. The correction is in that dictionary's
changelog rather than edited over the original, because silent redefinition makes
before-and-after comparisons meaningless.
What an agent should do differently
-
Record a metric's definition and collection method before collecting it. A definition
written afterwards describes what you got.
-
Write the limitation at the same time as the value. It is hardest to write honestly once
the number looks good.
-
Never report a test as passing unless it ran and passed. Name the checks that were skipped
or unavailable, and what that leaves unverified.
-
When two records disagree, publish the disagreement and investigate whether definitions,
populations or measurements changed. Do not pick the better number.
How this is most often got wrong
The tidy summary. Several findings collapsed into one confident sentence,
with the confounders left in a document nobody reads. Every experiment on this site has a
limitations section, and they are the most useful part of each page.
Reporting only what was measurable. An experiment that publishes its
available metrics and omits the rest has published a filtered result without saying so.