Shadow operation evaluates a candidate alongside real work while its proposed decisions do not determine the operational outcome under study. Live requests are copied to it, and its responses are logged or discarded rather than returned to the caller.
The definition concerns decision authority. It says nothing about what the candidate can reach, and the gap between those two is where shadow deployments cause the incidents they were supposed to avoid.
The boundary is not implied by the name
The word shadow does not establish that a candidate cannot write, send, call tools, consume shared resources, or store sensitive inputs. Those are properties of the deployment, and each needs inspecting: credentials, writable destinations, downstream calls, caches, quotas, and logging.
Then the boundary gets tested before anything relies on it. Disabling operational actions and verifying they are disabled are different events.
Monitoring covers the production service as well as the candidate, because candidate responses can be hidden while resource contention or unintended writes still affect the environment. Provider capabilities also carry exclusions — a shadow feature supported on one endpoint configuration is not supported on all of them.
Shared state contaminates the comparison
Traffic teeing creates interaction between deployments, and shared caches are the documented case.
Take a shadow variant receiving copied requests against a cache populated on misses. Those copied requests warm the cache, and later response times improve in both variants. The candidate now looks fast, or looks equivalent, and neither reading can be attributed to the candidate without accounting for the interaction. Discarding the candidate’s responses does not remove the cache effect, because the effect was caused by the request, not the response.
The related trap is the copied request itself. A request copied without the same permitted state is a different task, and every disagreement it produces is a difference in inputs rather than a model defect. Document the difference instead of interpreting it.
What to record
The workload, the fraction of requests copied, the context available at execution, and the comparison version. Stable request identities and timing. Duplication failures, candidate timeouts, and omitted cases.
Score against a checked rubric and use the actual capture denominator rather than the request count. Where responses are discarded, state which quality claims can no longer be assessed, because that decision is irreversible once the data is gone.
The thing it cannot measure
Operators did not act on any of these recommendations. So the study establishes nothing about their trust in the system, their workload, the quality of their review, or whether their decisions would change.
That is not a limitation to note in passing. It is the entire human half of the question, and it stays unobserved until a separate exposure decision is made with its own evidence plan. A system that performs well in shadow and is ignored in practice has passed the test it was given.
Cohorts, when exposure starts
Once people are involved, the selection questions arrive. Recruitment guidance identifies scheduling, location, activity, and recruitment route as sources of selection bias, and recommends varied approaches including people with access needs rather than the conveniently available.
Six populations get collapsed into one number: eligible, invited, enrolled, active, withdrawn, excluded. A pilot inviting 20 staff, enrolling 12, and observing 9 active users that reports satisfaction among the 9 has said nothing about the 3 who enrolled and did not use it, the 8 who declined, or anyone never invited. If all 9 work day shifts, overnight operations are untested.
Random assignment within volunteers supports the assigned comparison without making the volunteers representative. Inference and generalisation are assessed separately.
Progressive autonomy is not automatic
Moving from shadow to visible assistance to action is a sequence of separate decisions, not a maturity path. More users, more traffic, more tool access, and less human review are four different changes with four different evidence requirements.
Each increase in authority is a new bounded claim. A system that drafts internal notes reliably has not demonstrated that it can send them: recipient selection, permitted disclosure, send retries, and whether a reviewer can stop delivery are effects the drafting evidence never touched.
Stopping future actions and undoing completed ones are also different capabilities, and read-only access still exposes confidential information, so scope does not reduce to a read-write ladder. De-escalation stays available: holding or reducing authority is a legitimate outcome of a review rather than a failure of one.
The rule
What stays fixed is that shadow evidence covers outputs and runtime behaviour within a verified boundary. What changes is everything about people, and that requires exposure to observe.
Not to be confused with
A canary. A canary serves real users with real consequences at limited scale. Shadow withholds the output. The evidence and the risk differ accordingly.
A promotion rule. No request fraction, run duration, or automatic promotion threshold follows from any of this. Promotion is a justified exposure decision with its own plan.