How is RAG different from ordinary document search?

Ordinary search returns a list of documents for you to read. RAG returns a formulated answer plus a link to the fragment it was built from. Search is still underneath, but the result is an answer with grounds instead of ten links.

How often should the knowledge base be updated?

At the same rate the documents change. Technically this is handled by incremental updates of changed documents and index versioning, so “once a quarter” and “within the hour” differ by pipeline configuration, not by architecture.

Does the data leave for an external model?

It depends on the deployment, and that is decided at the start rather than at the end. Options range from a fully local model to an external provider with limits on which fragments may leave the perimeter at all. The choice is dictated by data requirements, not by fashion.

How many documents make this worthwhile?

Worth comes from access frequency and the cost of an error, not from volume. A hundred policies consulted daily pay the system back faster than ten thousand archived files nobody opens.

Can the answers be trusted without human review?

In sensitive domains, no, and that is an architectural decision rather than verbal caution. The system must show the source, refuse on empty retrieval and log what the answer was built on. Trust is then widened for specific question types, not all at once.

Applied AI · deep dive

What RAG actually is and when a business needs it

RAG means the model answers from your documents rather than from the internet, with a visible source under every claim. What it is made of, where quality is lost in chunking and updates, how to measure that it works and when you do not need it.

August 23, 20268 min read

What RAG is, in plain words

RAG is a “retrieve first, answer second” pairing. On a question the system first pulls relevant fragments from your documents, then asks the model to answer using only those. The answer comes with its source and can be checked.

The value lies in verifiability. Without RAG a model answers from training memory and sounds equally confident when right and when wrong. With RAG it shows where a statement came from, which turns it from a conversationalist into a tool.

RAG or fine-tuning

These solve different problems and are not competitors. Fine-tuning changes behaviour: style, format, stable domain rules. RAG supplies knowledge that changes: policies, prices, contracts, yesterday’s minutes.

A practical rule: if the answer must change the same day the document changes, it is RAG. If what changes is the manner of answering rather than the knowledge, it is fine-tuning. “The model must know our documents” almost always means the former.

What a working system is made of

Five parts, and quality is decided not by the last one but by the first two.

Sources. PDFs, wiki, mail, tickets, databases, spreadsheets. Each type has its own failure: PDFs bring tables and columns, wikis bring stale pages, mail brings quoted threads.
Chunking. A document is split into the fragments that will be searched. Fragment size and boundaries matter more than the choice of model.
The index. Vector search by meaning plus plain full-text search by words: codes, part numbers and contract numbers are found by words, not by meaning.
Ranking. A wide first pass, then a reranker picks the few best fragments. Without this step the model receives noise and answers from it confidently.
Generation with citations and the right to say “I don’t know”. An empty retrieval must lead to a refusal, not to elegant invention.

Chunking: where quality is usually lost

A 200-word fragment loses context: a clause without the section it belongs to means something else. A 2000-word fragment blurs meaning and search stops telling neighbouring topics apart. The working approach is to split along document structure rather than character count: section, clause, table, keeping parent headings alongside.

Tables and lists are extracted separately and kept whole: a table cut in half turns numbers into garbage. Every fragment carries metadata: document, version, date, owner, access rights. Filtering and the explanation of where an answer came from both rest on it.

Updating without downtime

A knowledge base is alive: documents are edited daily. Reindexing the whole corpus on every change is needlessly expensive, and answers from yesterday’s version devalue the system. The practical answer is incremental updates for changed documents plus index versioning: a new version is built alongside, the switch is atomic, the old one stays for rollback.

A deletion path is needed separately. When a document is withdrawn its fragments must disappear from retrieval the same day, otherwise the system will quote a cancelled policy with full confidence.

Access rights

The most expensive mistake in enterprise RAG is not a hallucination but an answer an employee should not have seen. Rights are enforced at retrieval, not at display: the permission filter is applied to the candidate set before fragments reach the model. Otherwise rephrasing the question is enough to get restricted data as a paraphrase.

How to tell the system works

The feeling that “it answers well” does not scale. You need a set of a few dozen real questions with known correct sources and three metrics: the share of questions where the right fragment made it into the candidate set at all; the share of answers whose citation supports the claim; the share of honest refusals where the documents hold no answer.

The first metric is fixed by retrieval and chunking, the second by ranking and prompting, the third by rules for empty retrieval. Without that separation improvements become random trial and error.

When you do not need RAG

If there are few documents and they fit in the model’s context, you do not need retrieval: put them in the request. If the questions are always the same, prepared answers are cheaper. If the documents contradict each other and nobody knows which version is current, RAG will not fix that; it will make it visible and fast, so start by putting the sources in order.