RAG vs fine-tuning: what should a knowledge assistant use?
If an assistant must answer from changing documents, start by checking retrieval and sources. If the problem is consistent output format or behaviour, test instructions and then consider fine-tuning. RAG and fine-tuning can work together; the choice depends on the failure you need to fix.
5 min read
How the approaches differ
RAG adds retrieved source passages to the answer context. Lewis and colleagues describe generation with access to an external retrieval index. A business assistant therefore needs document processing, retrieval and checks on the answer's evidence.
Fine-tuning adapts a model on training examples by changing trainable parameters. Full fine-tuning and adaptation methods such as LoRA differ in structure and cost. LoRA trains additional matrices while freezing the original weights. Training alone does not create a current-document citation or enforce access permissions.
These options are compatible: an adapted model can use document retrieval. First test a straightforward instruction-based baseline. If it meets the requirements, additional training may be unnecessary.
Choose according to the failure
Locate the failure in a specific answer. If the right document was not retrieved, model training does not repair retrieval itself. If the source was found but the answer violates the required format, index changes may not fix behaviour. Separate diagnosis helps avoid working on the wrong component.
| Need | What to check in RAG | When to test fine-tuning |
|---|---|---|
| Current internal facts | Current version reaches the context | If measurable behaviour failures remain with correct context |
| Evidence for an answer | Citation matches a retrieved source | To improve use of context while retaining citation checks |
| Consistent format | Whether instructions and structure validation suffice | Good examples and a separate evaluation set exist |
| Different access rights | Retrieval is limited to permitted documents | Training is not an authorisation mechanism |
| Terminology and response rules | Sources or instructions explain the terms | A recurring failure can be tested on unseen examples |
Test updates with a simple example
Illustrative example, not a client case: an internal request-handling document changes the responsible department. Keep the old and new versions, ask a question after updating the source and check that the assistant uses the new version. Also test a question the document does not answer.
Replacing a file is not enough. Check ingestion, the index, caches and the actual answer context. An old passage should not silently reappear from another source. Deleting a document also requires a check that retrieval no longer returns it.
Fine-tuning does not remove the need to distinguish current and outdated facts. If restricted data was used for training, retrieval permissions alone are not a sufficient protection. The access plan must also account for training data.
Compare options on one evaluation set
Define questions and acceptable outcomes before configuring the system. Include updated documents, missing answers, conflicting versions and access restrictions. Record an instruction-based baseline, then compare alternatives on the same separate evaluation set.
Measure retrieval of the required source separately from answer quality given that context. Manually inspect citations on selected examples: a citation's presence does not establish that it supports the claim.
Include latency, cost per accepted answer and update effort. Fine-tuning needs example preparation and evaluation after training; RAG needs source processing and retrieval maintenance. A cheaper option is still unsuitable if it fails acceptance.
| Scenario | Expected outcome | Evidence to inspect |
|---|---|---|
| Question about a current document | Answer from the current version | Version, passage and answer consistency |
| Missing fact | Stated limitation or human handover | No invented fact or source |
| Restricted document | Available data does not expose restricted content | Retrieval results, context and final answer |
| Required format | Answer meets structure and content requirements | Schema and semantic-field checks |
What can be applied to your project
Soudani and colleagues compare RAG and fine-tuning on questions about less-popular facts, finding an advantage for RAG in their experimental setting. This supports testing retrieval, but does not predict results on your documents or costs.
The matrix above proposes an engineering evaluation sequence. A project assessment needs sample documents and questions, update frequency, evidence requirements and access rules. With these inputs, architecture and pilot scope can be discussed.