Acceptance for a non-deterministic system evaluates whether it meets specified requirements despite output variation. The instinct to fix the variation first fails: a fixed seed makes a best effort to return the same response and guarantees nothing, and a change of model or parameters alters responses even with the seed held constant.
So the criteria move from the output to the properties of the output, and from a single run to a declared sampling rule.
Split deterministic from variable
Not everything varies, and exact equality remains correct where it applies. A required identifier, a schema, a mandatory field, a numeric total, an absent prohibited string — all are exact tests and should stay exact.
For the variable part, define acceptable task outcomes, prohibited effects, and the cases that count as unresolved. The failure to avoid is accepting any plausible paraphrase, which converts the criterion into a judgement about fluency.
The sampling unit is the whole argument
Four things get confused and need separating: baseline correctness, variation within a single case across repeated calls, coverage across different cases, and change over time.
Repeated calls on one fixed case measure the second. They say nothing about the third, and many calls sharing one case are not independent evidence about the population. Neither are many results collected during one outage.
Uncertainty is therefore assessed at the unit actually sampled. Confidence intervals for proportions are the standard tool, with exact binomial methods available when failure counts or samples are small — and the binomial reference assumes a fixed trial probability, which is an assumption about the application that has to be examined rather than inherited.
Freeze the configuration and the intended population before starting. Specify cases, repetitions, sampling unit, rubric, uncertainty method, and stopping rule before seeing results. Retain every valid result, distinguish harness failures from system failures, and exclude only under the rule declared in advance.
Where human or model graders produce the verdicts, their error and disagreement are part of the measurement rather than the ground truth against which everything else is measured.
What the contract has to say
Acceptance relates to the applicable contractual quality and quantity requirements, subject to the other terms, and quality assurance precedes it in the normal case. Timing is a choice: before, at, or after delivery.
The federal services inspection clause is a useful model for the remedy structure, because it distinguishes reperformance from defects that cannot be corrected that way. That distinction matters for a system whose behaviour is statistical, since “do it again” is not a remedy for a distribution.
An acceptance schedule therefore defines the deliverable, the test version and environment, the evidence, the decision authority, the exceptions, the cure and retest process, and the consequences of failure.
The distinction that saves projects is between accepting a deliverable and approving a system. A discovery contract promising a documented evaluation and reproducible evidence, with no promise that the candidate meets an operating threshold, is conformed to by an evaluation showing the threshold was missed. The buyer assesses the research deliverable and separately declines the launch. Merging the two forces a false choice between paying for work that was performed and deploying a system the evidence argues against.
Warranties, and what as-is does not do
Warranty terms specify the warranted subject, the obligations, the remedies, the scope, and the duration, with notice provisions to match.
As-is wording operates as a contextual exclusion of implied warranties under the relevant commercial code, with limits on language contradicting an express warranty and a separate consumer-goods qualification. So an agreement combining as-is wording with a specific promise that delivered code will build in a listed environment contains an express promise, and a failed build is assessed against that promise rather than dismissed by the label.
Where a prototype’s users, data, or consequences change, the promises have to be rechecked against the new operation. Support commitments, acceptance requirements, liability allocation, and third-party dependencies are four separate clauses, and one of them does not answer for the others.
The rule
What stays fixed is that acceptance requires a predeclared decision rule applied to a declared population. What changes is the threshold and the sample, which follow the consequences of being wrong for this specific use.
Not to be confused with
Reproducibility. A system can be acceptable and non-reproducible. Pursuing exact repetition as a precondition delays acceptance indefinitely and buys nothing the criteria need.
A right to reject. An acceptance schedule reconciles with the executed contract. It creates no unilateral right to withhold payment or impose tests the parties did not agree.