The decoder scored 100 percent. The encoder, on the same format, by the same method, scored 8.3 percent.
Both were reconstructed from observed input and output behaviour alone. Both passed everything they were built against. One held a correct model of the mechanism and one did not, and the only reason that is knowable is that the real algorithm sat sealed and hash-verified in the repository the whole time, opened only after the reconstruction was frozen at a tag.
This is the fourth of four experiments in AI-assisted reconstruction of legacy systems. The first three degraded the source of truth: a scanned specification, then a novel, then the memory of a product. This one removes it. There is no specification and no source, only behaviour, plus an oracle that answers any question about outputs. It is the only configuration in which two questions become measurable rather than assertable.
The first is whether behavioural match equals understanding. A reconstruction can pass every test while holding a wrong model, with untriggered modes waiting in the part of the input space never sampled. The second is whether a strong oracle conceals that. A model fits the observed outputs while inventing a plausible and wrong internal mechanism, and because the tests stay green the error is rewarded rather than exposed.
Choosing a target the model has not memorised
The target is the -lh5- method of the LHA archiver: LZSS with an 8 KB window plus static Huffman coding.
Four properties made it suitable. The original runs, so the oracle is unlimited. The algorithm is public, so a real answer key exists and confabulation can be graded rather than suspected; a never-solved black box demonstrates the hazard exists but never quantifies it. The method is bounded at a few hundred lines. And the encoder heuristics are underdetermined by the output: match-finder strategy, minimum match length, tie-breaking, greedy versus lazy matching and block splitting are encoder-side choices that leave no trace in the decompressed bytes.
A more famous archiver would have been the wrong specimen. Widely-discussed formats are heavily memorised, so the exercise would measure recall rather than reconstruction. The general LZSS and Huffman scaffold is in every model’s training data; the specific choices in this implementation are not, and the specifics are what gets measured.
Three controls
The seal. The original source sat git-ignored and hashed. At unsealing, verification confirmed 107 of 107 files intact and the manifest’s own digest matched the value pinned in the sealing document. Reading it before the freeze is the single act that would have destroyed the measurement, so detection was made mechanical.
The freeze. The reconstruction was tagged and committed before unsealing, so no artifact could be adjusted to fit the answer once the answer was visible.
The cold-recall document. Before any reconstruction began, a model recorded everything it already believed about the target. Without it there is no way to separate derivation from memory, and the exercise collapses into a memory test.
Three models split the work deliberately: one reconstructed the decoder, a second the encoder and the cold-recall baseline, a third ran the design thread.
The decoder recovered the format, and it credits memory
Nineteen of nineteen round-trips exact.
The sharpest single result sits here. The implementation’s generic macro defines a 5-bit position-table field width, so a source reader’s first guess is wrong for this method, which uses 4. The reconstruction recovered the method-specific value from round-trip behaviour alone, against a constant that would have misled someone reading the source.
Checking the rest against the cold-recall document deflates it. Nearly every format fact the decoder recovered was already in the committed prior: the window size, the match thresholds, the alphabet size, the precode and its zero-run encoding, the block count width, the position-slot decode. The perfect score credits prior knowledge rather than derivation, which is unsurprising: the format is one-way determined, and memory and derivation converge on the same answer.
The encoder did not recover the choices
Byte-match against the real implementation, on a fresh-seed corpus the reconstruction never saw:
| Input class | Trained on | Byte-match |
|---|---|---|
| Text | Yes | 0 of 3 |
| Source code | Yes | 0 of 3 |
| Structured | Yes | 0 of 3 |
| Repetitive | Yes | 1 of 3 |
| Random | No | 2 of 2 |
| Edge cases | No | 3 of 5 |
| Trained total | 1 of 12 (8.3%) | |
| Held-out total | 5 of 7 (71.4%) |
A perfect decoder coexisting with an 8 percent encoder states the finding in one line. The format was recoverable. The encoder’s choices were not.
The predicted overfitting curve inverted
The plan expected high scores on trained classes falling on held-out ones. Held-out scored 71 percent and trained scored 8.
The split tracks difficulty rather than exposure. The trained classes are the ones that exercise real compression; the held-out set is dominated by incompressible and trivial inputs that fall through to stored mode. Random data scores 100 percent because no compression heuristics run at all.
The honest statement is therefore not a generalisation gap. Byte-identity is reached only where the encoder heuristics never fire, and wherever genuine compression happens the reconstruction diverges. Rates were identical on the trained-seed and fresh-seed corpora, so the divergence is systematic rather than instance overfitting.
The confabulation catalogue
Graded against the now-visible source, the tally is six decisions recalled correctly, five derived correctly, one wrong but passing, two never triggered, and one visible divergence.
Two are clean derivations. Nearest-offset tie-breaking and one-step lazy matching are both recorded as unknown in the cold-recall document, in its own words, and both were recovered from black-box behaviour.
A caveat inverts on inspection. The same model wrote both the recall document and the encoder, which ordinarily weakens an independence claim. Here it strengthens it: a model cannot conceal prior knowledge from its own committed statement of ignorance. Deriving both heuristics after recording that it did not know them is the strongest evidence in the run.
The failure concentrates in one place. The original builds a frequency heap and assigns code lengths in the order symbols leave the heap, governed by the exact comparison semantics of its sift-down operation rather than by symbol value. Textbook canonical Huffman breaks ties by symbol index, producing different lengths for equal-frequency symbols, which cascades through the entire bitstream. The reconstruction localised the problem correctly and never replicated the ordering.
The oracle did not mask the error
The experiment was built to demonstrate that a strong oracle conceals confabulation. It largely refuted that.
The central error is a visible divergence. It fails the byte test loudly, and the oracle caught it. Only two gaps stayed hidden: a block-splitting path requiring inputs larger than 8 KiB to fire, and an unmodelled limit in the match finder that needs input more self-similar than anything tested.
Both are artifacts of a corpus that tops out at 8 KiB. The masking risk is therefore not a property of the oracle’s strength but of the input space sampled. A strong checker over a narrow corpus conceals precisely the mechanisms the corpus never exercises, and conceals them silently, because everything it can see is green. Strengthening the checker changes nothing. Widening the corpus does.
That generalises past compression. It is the same failure as a test suite with excellent coverage of the paths someone thought to write a test for.
What the measurement cost
The apparatus producing those graded results — the pinned oracle, the deterministic corpus generator, the seal and its manifest, the cold-recall protocol, the freeze tag, the grading harness — is most of the repository. The reconstruction is the small part.
That ratio is the transferable lesson. Establishing whether a reconstruction understood a system rather than merely passed requires building the apparatus that can tell you, before starting. Afterwards, every artifact is contaminated by the answer.