Synthetic, masked, sampled, and production name the data a prototype runs on. They are routinely presented as four rungs of one ladder, from safest and least useful to riskiest and most informative. They are not.
Synthetic, masked, and sampled are construction choices, and they overlap: a sample can be masked, and synthesis can be fitted to a sample. Production describes an operating context rather than a construction. Ordering four things that are not the same kind of thing produces a ladder nobody can actually climb.
Fidelity is relative to a claim
Fidelity is preservation of the properties needed for a particular inference or test. It is not general realism.
The de-identification literature makes the point precisely: data accuracy and utility for a specified purpose are different, accuracy for one computation does not establish accuracy for another, and increasing privacy protection does not reduce utility for every use. The assumed straight-line tradeoff between privacy and usefulness does not hold as a general law.
So the question is never which dataset is most realistic. It is which properties would change the decision, and whether this dataset preserves them.
Synthetic
Synthetic data is artificially constructed, ranging from designed examples to generation fitted to source data. Official guidance treats it as artificial and warns that it will not preserve every property of the source, with intended use guiding construction and disclosure checks preceding wider release. Randomly selected original rows are not synthetic data.
The privacy assumption is the dangerous part. Experiments on tabular generators found privacy gain varying across selected target records, generators, and attack features, with some outlier targets remaining vulnerable to membership and linkage inference. That result counters automatic privacy claims for the tested generators without formal guarantees. It is a counterexample rather than a ranking of generators or a proof that useful private synthesis is impossible.
What synthetic data reliably supports is structural testing. A mortgage test set with valid fields and plausible values exercises a parser on the structures it contains. Where it omits joint applicants and unusual income combinations, it evidences nothing about those, and adding designed edge cases improves coverage without establishing how often they occur.
Masked
Masking transforms selected values. De-identification is assessed against a specified disclosure objective and context, and the two are not the same operation.
Field masking can remove direct identifiers while leaving quasi-identifiers insufficiently treated, which is why masking alone is not advised for public release or as the sole confidentiality mechanism in non-public sharing. A tool that handles one data modality can miss identifying information in another, and free text is the recurring example.
Masking also breaks things silently. Remove names and replace customer identifiers independently in two tables, and the joins fail. Repair the mapping so joins work, and a linkable identifier now exists that itself needs protection. Neither state establishes anonymity, and an unprocessed support-note field remains a separate exposure through all of it.
Under GDPR, pseudonymisation is defined in terms of additional identifying information held separately and protected. Pseudonymised information remains potentially attributable and is distinguished from anonymous information, with identification judged by the means reasonably likely to be used.
Sampled and production
Random sampling is designed to represent process variation, and cannot guarantee that any particular realised sample is representative. Existing operational records do not escape this: coverage, missingness, invalid-data edits, failed linkage, and collection and processing errors are quality questions for administrative data as much as for surveys.
The failure is selection rather than volume. An evaluation extracting only successful transactions from one recent weekday produces a high completion score that characterises neither failures, nor weekends, nor excluded customers. A deliberately enriched failure set is useful for debugging and does not estimate how often failures occur — those are two purposes and they need two datasets, labelled.
The rule
What stays fixed is that every result inherits the limits of the data that produced it. What changes is which limits apply, so each result carries a fitness statement: dataset version, how it was constructed, which properties were tested, the intended population, the access context, and the conclusions it does not support.
Those statements travel to the receiving team. A prototype that passes format tests on designed examples, then tests joins on a masked extract, then evaluates a defined operational sample has three results supporting three claims. Collapsing the sequence into a production-readiness score discards all three.
Where the strategy fails
Privacy is inferred from construction. Neither synthesis nor masking establishes a disclosure property. Both need evaluation against a stated adversary and the auxiliary information actually available.
Fidelity is inferred from realism. A dataset that looks like production and omits the relationship under test is worse than designed examples that contain it, because it invites a conclusion it cannot support.
Not to be confused with
A release authorisation. Source rights, generation permission, and the legal standard applying to each recipient are separate determinations from whether the data is fit for the test.
Lower controls. A closed environment and non-production data are relevant facts. Neither establishes every exclusion nor removes vendor and security concerns.