Every de-identification transform removes signal, and some of that signal is what the system under test was going to use. The pilot then measures a system solving a different problem from the one it will face.
This is knowable in advance rather than discoverable afterwards, which is what makes it a rule rather than a caveat.
Start from the relationships, not the columns
Before choosing transformations, list the relationships the prototype must actually test. Then check each transform against that list: whether replacement values preserve joins, uniqueness, ordering, formats, and the correlations the model depends on.
The failure is mechanical and reproducible. Remove names and replace customer identifiers independently across two tables, and the joins break — the system now fails for a reason that has nothing to do with its design. Repair the mapping so the joins work, and a consistent linkable identifier now exists across the dataset, which is itself an exposure requiring protection.
Neither state is anonymity, and the second one feels like success.
Masking is not de-identification
Field masking transforms selected values. De-identification is assessed against a specified disclosure objective and context. Treating the first as achieving the second is the standard error.
Masking can remove direct identifiers while leaving quasi-identifiers insufficiently treated, which is why it is not advised alone for public release or as the sole confidentiality mechanism in non-public sharing.
The most-missed exposure is modality. A tool that handles structured columns correctly can miss identifying information in free text, and the guidance calls for validation on the actual data types present rather than on the ones the tool was configured for. An unprocessed support-note field survives every column-level transformation in the pipeline.
Under GDPR, pseudonymisation is defined in terms of additional identifying information held separately and protected. Pseudonymised information remains potentially attributable and is distinguished from anonymous information, with identification judged by the means reasonably likely to be used. Where identifying mappings exist, who can access them and under what authority is a separate decision from whether the transformation ran.
Synthetic data does not solve it either
Synthesis fails on the same axis from the other direction. Official policy treats synthetic data as artificial and warns that it will not preserve every property of the source, with intended use guiding construction.
And the privacy assumption does not hold automatically. Experiments on tabular generators found privacy gain varying across target records, generators, and attack features, with some outlier targets remaining vulnerable to membership and linkage inference. Absence of exact copies is not evidence of privacy.
What synthetic data supports reliably is structural testing on the structures it contains. A test set omitting joint applicants and unusual income combinations evidences nothing about those, and adding designed edge cases improves coverage without establishing their frequency.
The tradeoff is not a straight line
The assumed inverse relationship between privacy protection and usefulness does not hold as a general law.
Data accuracy and utility for a specified purpose are distinct concepts. Increasing privacy protection need not reduce utility for every use, and accuracy for one computation does not establish accuracy for another.
So the question is never which dataset is most realistic. It is which properties would change the decision, and whether this dataset preserves those.
A dataset that resembles production and omits the relationship under test is worse than designed examples that contain it, because it invites a conclusion it cannot support while looking like it can.
Selection contaminates alongside transformation
Treated data carries the sampling problems of the extract it came from, and those compound.
Random selection is designed to represent process variation and cannot guarantee that a particular realised sample is representative. Coverage, missingness, invalid-data edits, failed linkage, and processing errors are quality questions for administrative records as much as for surveys.
An evaluation drawn from successful transactions on one recent weekday, then masked, has two independent reasons its results do not generalise, and the masking discussion consumes all the attention.
The rule
What stays fixed is that a result inherits the limits of the data that produced it. What changes is which limits apply, so each result carries a fitness statement: the dataset version, how it was constructed, which properties were tested, the intended population, the access context, and the conclusions it does not support.
Those statements travel to the receiving team. A prototype that passed format tests on designed examples, tested joins on a masked extract, and evaluated a defined operational sample holds three results supporting three claims — and collapsing them into a readiness score discards all three.
Not to be confused with
A release authorisation. Source rights, generation permission, and the legal standard applying to each recipient are separate determinations from whether the data fits the test.
Lower controls. Reduced fidelity is not permission to relax handling. A closed environment and treated data are relevant facts that establish neither every exclusion nor the absence of vendor and security concerns.