Prompting changes the supplied instructions, examples, or context. Retrieval selects external material for use during a request. Fine-tuning changes trained parameters through an adaptation process. These are three intervention points in one system, and the recurring error is treating them as three competing strategies.
They coexist by design. The original retrieval-augmented generation work paired a retrieved index with a generator and fine-tuned the query encoder and generator while keeping the document encoder and index fixed — retrieval and parameter adaptation in the same architecture from the start. Later work combines supervised adaptation with test-time retrieval directly, training on examples that include relevant and distractor documents and omitting the relevant document for some of them. That is a demonstrated combined design rather than a universal recipe, and a comparator beats it on at least one task in its own results.
Diagnose before intervening
A failure has a location, and the intervention follows from it. Four questions, in order.
Did the needed evidence exist in the accessible corpus at all? If not, this is corpus maintenance, and no amount of prompting or training substitutes.
Was it retrieved? If the document exists and the retriever did not return it, the problem is indexing, chunking, or ranking.
Did it reach the actual request? Retrieval returning a document and that document surviving truncation into the rendered prompt are separate events.
Was it used correctly once present? Only this last one is a candidate for prompting changes or adaptation.
A policy assistant that starts failing after the source policy changed is answerable by walking those four in sequence. Training a new model before tracing the failure is an untested choice with a large bill.
Position is a variable
Where evidence sits in a long context changes performance on the tested tasks and models — the experiments reorder documents while holding the intended answer constant, and the results move.
This makes context arrangement an experimental input, not a packing detail. It does not license a universal rule about putting the answer first: the finding is that order matters, which means order has to be tested rather than assumed in either direction.
Comparing them honestly
Hold the task scope fixed and report changes to model, prompt, corpus, and scoring separately. Include a baseline without the proposed addition.
For each option the evidence package differs. Prompt changes need a versioned template, a rendered-input policy, and task results. Retrieval changes need corpus provenance, permissions, retrieval results, and answer evaluation. Fine-tuning needs training provenance, configuration, the resulting model identity, and held-out results.
The maintenance exercise differs too, and it is the part that decides transferability. Can the recipient edit, test, and release a prompt change? Update or remove a source document and verify the downstream behaviour? Reproduce or replace the adaptation workflow under the rights and compute actually available to them?
Evaluate removals as well as additions. A system that answers well and cannot forget a withdrawn document has an unsolved problem that initial-answer testing never surfaces.
Selection and lifecycle
Model selection is revisitable, and the unit of comparison has to be stated. Comparing a tuned application against an untouched baseline answers a system question and cannot isolate the model’s contribution. Holistic evaluation work is explicit that evaluating a model and evaluating a scenario-specific system are different exercises.
The selection record separates constraints that must be met — authorised deployment, required features, acceptable failure types, response-time limits — from tradeoffs to be measured. Missing evidence stays missing; a candidate does not earn a favourable default for an unmeasured dimension. A candidate leading a general benchmark while lacking a required deployment option has not won, and extra benchmark points do not erase the constraint.
Lifecycle evidence attaches to the ownership horizon. Published guidance separates functioning integration code from the quality of model responses, so a replacement is not an endpoint rename. Where a retirement is announced inside the horizon, the replacement plan is part of the decision.
What versioning has to capture
The distinction that matters is between a template, a rendered request, and a release. Registries offer immutable versions alongside mutable aliases and changeable tags, and only the first is an identifier.
The run manifest records the resolved prompt identity, instruction order, template variables, model and endpoint, generation settings, tool definitions, retrieval index revision, selected document identifiers and versions, ordering and truncation rules, output schema, and scoring version. Where an alias was used, preserve the version it resolved to at the time.
A deployment keeping an alias called production, pointed at a new version, with a changed retrieval index, has two moving parts and one name. The manifest separates them so the comparison can be designed; the alias alone cannot identify what an earlier request used.
Retention is bounded by what is authorised. Record what cannot be reconstructed and how that limits later diagnosis, rather than retaining every user input in pursuit of completeness.
The rule
What stays fixed is that each intervention has its own evidence package and its own maintenance burden on the receiving team. What changes is which one the diagnosis points to, and the diagnosis has to be done first.
Not to be confused with
A rights solution. No technique settles data rights, privacy, or factual correctness by its name. A retrieved document is not thereby a verified source or a faithfully used one.
A one-time decision. Corpus, model, task, access, and workload all change, and each change reopens the comparison.