Four reconstructions, four kinds of missing documentation, and the defects landed in the same place every time.
In the flight-language rebuild it was the characters the scanner destroyed. In the submarine simulator it was the parameters the source never stated. In the encyclopedia revival it was the desktop window no headless harness can open. In the compressor it was the part of the input space the corpus never sampled.
Different systems, different languages, different symptoms. One shape: the defects clustered wherever no automated check could reach, and in every case the code looked correct. Tests green, types satisfied, output plausible.
What was varied
These were not four projects that happened to involve old software. They were one experiment with a controlled factor: how degraded is the source of truth, and can the model be caught being wrong.
| Reconstruction | Source of truth | Oracle | Outcome |
|---|---|---|---|
| Flight language | Degraded formal specification plus preserved code | Strong, independent | Faithful; all twelve preserved programs run |
| Submarine simulator | A novel | None; one was built | Five findings, each bounded |
| Encyclopedia revival | Memory of a product | None; human judgement only | Shippable, every defect at the runtime seam |
| Compressor | Behaviour only, key sealed | Strong, opened after freeze | Decoder 100%, encoder 8.3% |
The expectation was that quality would degrade smoothly as the source got worse. It did not. A different factor predicted the results, and it cut across all four.
Determined beats difficult
Reconstruction quality tracked whether the available evidence determines the answer, not how hard the task was.
The compressor makes this measurable, because both halves ran under identical conditions. The decoder recovered the bitstream format completely, because a format is one-way determined: exactly one answer decodes the stream. The encoder scored 8.3 percent on the same format, because its heuristics are choices that leave no trace in the decompressed bytes. Same model family, same oracle, the same week. A twelvefold gap, explained entirely by whether the evidence pinned the answer.
The other three restate it in their own terms. The 485 grammar productions were determined by the page images and the reconstruction recovered them; the characters the scan destroyed were determined by nothing, and no increase in model capability retrieves them. The submarine’s physics is determined and produces the same crush depth for anyone, while the parameters the novel omitted leave submerged endurance spanning a factor of 35 and battery capacity a factor of 100 — spreads that are the finding rather than noise around it. In the encyclopedia, parsing and ordering and cache naming are determined by their inputs and compiled correct immediately, while whether the interface feels like the original is determined by nothing writable and stayed a human judgement throughout.
The practical form: before handing legacy work to a model, establish whether the evidence you hold pins the answer. Where it does, expect success and build the cheap mechanical check that confirms it. Where it does not, expect confident invention, and note that the model does not signal which mode it is in, because from the inside the two are identical.
A strong oracle is not the protection it appears to be
The compressor experiment was built to demonstrate that a strong oracle conceals a wrong internal model. It largely refuted that.
The central error failed the byte-comparison test loudly, and the oracle caught it. What the oracle did conceal were two mechanisms the corpus never triggered: a path requiring inputs larger than the test data, and a limit in the match finder needing input more self-similar than anything tested.
The masking risk is a property of the input space sampled, not of the checker’s strength. A strong checker over a narrow corpus conceals precisely the mechanisms the corpus never exercises, and conceals them silently, because everything it can see is green. Strengthening the checker does nothing. Widening the corpus does. That is the same failure as a test suite with excellent coverage of the paths someone thought to write a test for.
What the human contributed
Across all four projects, almost none of the supervision effort went into reviewing code.
In the flight language it was declining to take a majority vote between OCR engines, then reading 485 grammar productions against page images. In the simulator it was signing the invariants and making them append-only, so the model could repair the physics but could not weaken the checks that judge it. In the encyclopedia it was operating the real application on the real platform, where both defects lived. In the compressor it was holding the seal, and not reading the answer key until the work was frozen.
Two categories, and neither is code review. One is supplying what no code can check. The other is designing the checks that are possible and putting them structurally out of the model’s reach.
The OCR decision is the sharpest, because it separates a good result from a worthless one that looks identical. Running several OCR engines and taking the majority is the obvious move and it is wrong: the errors are correlated, so a vote launders error instead of cancelling it, and no fusion recovers a character no engine produced. Survivable in prose, fatal in a grammar where the metacharacters are the content.
The invariant signing is the most reusable. Twelve invariants live in version control behind a manifest checksum that only a human re-signs. When the fuzzer finds a violation, the repair workflow requires patching the system and forbids touching the invariant. An agent optimising against a check it can also edit is not being checked.
Build the means of checking before the thing being checked
Ranked by how much verification apparatus existed before the real work started, the four projects come out in the same order as how much their conclusions can be trusted.
The simulator had determinism, golden-stream replay and malformed-input handling green before a single equation was written, plus a frozen protocol and a rule that every parameter traces to either a cited claim or a documented gap. Its numbers are the most defensible. The compressor’s seal, cold-recall protocol, freeze tag and grading harness are most of that repository, with the reconstruction the small part. The flight language inherited an independent interpreter, which is a luxury and the reason it is the favourable case.
The encyclopedia had none of this available, and it shows: its conclusions are qualitative, argued from patterns across features rather than measured. They remain the most immediately useful findings in the set, and the least provable. Both facts follow from the same missing oracle.
One trap deserves naming. A verification harness that finds nothing is indistinguishable from a broken one. The simulator’s 24-hour soak ran 879,643 ticks and found zero invariant violations, which carries information only because the harness was first validated against a deliberately poisoned invariant, which it caught, minimised from seven commands to two, and packaged.
Three results that went the wrong way
A programme that reports only confirmations is a demonstration. These four produced three clean refutations, and they were the most informative outputs.
The compressor predicted the classic overfitting curve and produced its inverse, because the split tracked difficulty rather than exposure. It also predicted the oracle would conceal the confabulation, and the oracle caught the main error while a narrow corpus hid the rest. The simulator’s fuzzer was built to find physics defects and found none; the only finding in the loop was a flaw in the design of the experiment, where the optimiser had been maximising survival instead of performance because the controller pinned its own objectives.
Each changed a conclusion. None would have surfaced from reasoning about the problem.
Limits of this evidence
Four projects, one operator, no control group. There is no human-only reconstruction to compare against, so nothing here establishes that AI was faster or better than the alternative — only what it managed and what it did not. The projects were chosen partly for being interesting, which is a selection effect. Model versions moved during the work. Three of the four are Rust, which is a narrow language sample.
The missing measurement is a like-for-like human baseline on the same corpus. Producing one would require someone to transcribe 485 productions and reconstruct an encoder without a model.
What this means for a modernisation programme
Handing legacy code to a model works well enough that the useful question is where to spend your own attention.
Not on the code. The logic was correct nearly every time. Spend it at the host, runtime and operating-system boundary, where every defect in the desktop project lived and no harness reached.
On knowing which mode you are in. Where evidence determines the answer, build the cheap mechanical check: numbering integrity, reference integrity, a verbatim grounding gate. Where it does not, force the uncertainty into the open with a divergence log, a gap inventory carrying plausible ranges, an ambiguity recorded rather than resolved. Each of these projects has a file whose only job is holding what was not derivable, and those files are what make the rest credible.
On the checks the model cannot edit. Sign them, hash them, require a human to weaken them.
On widening the input space rather than tightening the oracle. The gaps that survive strong testing are exactly the ones your corpus never triggers.