An evaluation harness is the executable and documentary machinery for applying defined tests to a defined system and interpreting the results. It is a deliverable rather than a working artifact, because the system it measures can be rebuilt and the encoded definition of correct behaviour cannot.
That definition is expensive. It is accumulated from subject-matter judgement, disputes about edge cases, and every disagreement someone resolved during the build. Losing it means paying for those decisions again with different people and getting different answers.
What ships
Six components, and the packaging matters as much as the contents.
A scope manifest naming the decision the evaluation informs, the system revision, the included tasks, the exclusions, the maintainer, and the recipient. Test assets with case identifiers, inputs, expected outcomes or rubric, provenance, permission, and split membership. An execution specification covering dependency versions, environment, commands, configuration, how access is obtained, and randomness controls. Scoring as executable checks, plus human-rating instructions where they apply, a failure and skip policy, and the rationale for each threshold. A result bundle with per-case outcomes, errors, raw outputs where authorised, the aggregate calculation, run identity, and known limits. And a maintenance record with the change log, case retirements and additions, review triggers, owner, and approvals.
Live credentials and unauthorised datasets do not travel in a portable archive. Where assets cannot move, document the permitted access route or a replacement set, and state the coverage difference that creates.
The acceptance test is behavioural
The builder supplies the package and the receiving evaluator performs a clean run. The order of that run is what makes it evidence.
Start with harmless fixtures and confirm two things: that deliberately incorrect output fails, and that an unavailable dependency produces a visible run error. A harness nobody has seen fail is a harness nobody has tested.
Then run the authorised task set. Predefine which conditions make an unavailable dependency a harness error and which make it a failure of the system under evaluation — classifying the second as the first is how poor reliability disappears from a report.
Reconcile the case counts. Passed, failed, skipped, and invalid must all survive into the reporting. A package of 20 cases where 15 pass, 3 fail, and 2 cannot run for want of fixture access has a valid-case pass rate near 83 percent and a planned-case pass rate of 75 percent. Both are true, neither should be hidden, and the reviewer’s first question is why those two cases were unavailable and whether the exclusion matched the policy declared in advance.
For probabilistic systems, predefine how repeated runs are reported and what tolerance applies, rather than promising identical text.
Three failures the harness invites
Leakage. Information from the test set reaching model fitting or preprocessing inflates the estimate. Transformations are learned on training data and applied to test data; doing it the other way measures the harness.
Seed confidence. Reproducing an execution and establishing robustness to estimator randomness are separate questions. A good result under a fixed seed says nothing about other initialisations, and reporting it as though it does converts a repeatable run into a claimed property.
Silent scoring drift. A revised rubric changes scores without any change to the system. Preserve old results alongside the changed rubric, or the improvement is in the measuring instrument.
Reproducible, and still wrong
A package can be operationally reproducible while its cases are biased and its rubric is wrong. Those are three separate acceptance questions and they need asking separately: can the recipient rerun it, does it cover what matters, and does the result support the decision it is used for.
Keeping development cases separate from held-out decision evidence is the discipline that makes the third question answerable. Where a set has been reused during tuning, record the exposure rather than describing it as unseen.
The rule
What stays fixed is that the harness has an owner after transfer who can add cases, retire them, and explain a failure. What changes is the system under test, and a harness tied to one revision of it stops being usable at the first significant change.
Not to be confused with
A scientific evaluation. Packaging and ownership are what this covers. Whether the tasks, samples, and metrics support the conclusion is a separate body of work, and a well-packaged harness measuring the wrong thing transfers that error intact.
Monitoring. A harness answers questions on demand about a fixed set of cases. Watching a live service for drift is a different instrument with different data.