How to update a RAG knowledge base without stopping the system

Why updating is an engineering problem of its own

In a demo the corpus is loaded once and updating never comes up. In production documents change daily, and a system answering from a superseded revision is worse than no system: it sounds just as confident while quoting a cancelled rule.

So updating is designed together with retrieval rather than bolted on later. Three questions need answers before launch: how a document enters the index, how it leaves, and what a user sees at the moment the index changes.

Incremental updates instead of full reindexing

Reindexing the whole corpus takes hours and real money in embeddings, and there is no reason to run it because one document changed. The working pattern: a content hash at document and chunk level, recomputing only the chunks that changed and leaving the rest alone.

This has a non-obvious consequence: chunking must be deterministic. If the same document is split differently each run, the hashes never match and the “incremental” update becomes a full reindex under another name.

Swapping the index without a restart

Restarting the service to refresh the base is not a solution but a postponement: it drops open sessions and makes updates rare, because each becomes an event. The standard approach is a versioned index behind a pointer: the new version is built alongside, verified, then the pointer is switched atomically while the old version is kept until in-flight requests finish.

The crucial part is verification before the switch, not after. An index built with a broken pipeline switches over just as smoothly as a correct one and is discovered only through complaints.

Deletion: the part remembered last

A document is removed from storage while its chunks stay in the index, and the system keeps answering from something that no longer exists. In sensitive domains that is not an inconvenience but a breach: an employee sees data whose access has been revoked.

The minimum working scheme: the document store is the source of truth and the index is derived from it; deletions and permission changes travel the same pipeline as additions and are verified by the same test. Access rights live in chunk metadata and are applied as a pre-search filter rather than trimmed out of the answer.

What updating does to retrieval quality

Adding to an index changes not only its contents but the competition for the top results: new chunks displace old ones on similar wording. A separate trap is changing the embedding model: old and new vectors are not comparable, and such a move requires a full rebuild rather than an append.

• The share of questions where the right fragment reached the top results — before and after the update, on the same question set.
• The share of answers with empty retrieval: a rise means something fell off in the pipeline.
• The age of the freshest document in the index — the simplest indicator that updates are running at all.
• Update cost: how many chunks were recomputed per document edit.

What is enough at the start

Not every project needs real-time updates. If policies change quarterly, a nightly rebuild with verification before the switch closes the task entirely and costs far less to maintain.

The difference between “once a day” and “within the hour” is pipeline configuration, not a different architecture. The difference between having deletion and not having it, however, is architectural and must be built in from the start.