Checklist: are your documents ready for RAG
Fifteen checks we run before agreeing to build retrieval over a document corpus. You can answer them in an hour on your own and find out whether sensible answers are achievable on your data or whether the data needs work first. The full list is on this page; the PDF needs no sign-up.
- Fifteen checks in four groups: corpus, structure, permissions, verifiability.
- Each one explains what breaks if the answer is no.
- At the end: how to read your result and where to start in each case.
The corpus: what you actually have
- 01
You can name the exact place where the documents that need answering live.
“Somewhere on the drive and in Marina's inbox” is not a corpus. Until the source has one address the scope cannot be estimated even to an order of magnitude.
- 02
You know how many there are: hundreds, thousands or tens of thousands.
The order of magnitude decides everything: on a hundred documents almost any approach works; on tens of thousands chunking and retrieval strategy start to decide the outcome.
- 03
You know what share are scans and photos rather than text.
A scan means a separate recognition stage with its own cost and its own error rate. Discovering it mid-project always moves the deadline.
- 04
You can say which version of a document is the current one.
The system will find and quote both revisions with equal confidence. It can only tell the current one from the superseded one by a signal that exists in the data.
Structure: how a document is built
- 05
Documents have section headings rather than one continuous block of text.
Headings are the cheapest boundary to chunk on. Without them you cut by length, and half the answers will break off mid-thought.
- 06
You know whether the documents contain tables and how much the answers depend on them.
A table cut in half produces a confidently wrong answer: the row is there, the column header stayed in another chunk. Tables get handled separately or not at all.
- 07
You know whether documents cross-reference each other.
“In the manner set out in Appendix 3” — without Appendix 3 the answer is formally correct and practically useless.
- 08
Duplicates and near-duplicates are known at least approximately.
Five versions of one policy crowd everything else out of the results: retrieval reads them as five independent confirmations of the same thing.
Permissions: who sees what
- 09
You know whether every document may be shown to every employee.
If not, permissions must be designed into retrieval from day one. Adding them on top of a finished system means rebuilding the index.
- 10
You know whether the corpus contains personal data.
This decides where the corpus may be processed at all, which model is permissible and what has to stay inside your perimeter. It is an architecture question, not a legal one.
- 11
There is a person accountable for the content of the documents.
When the system quotes a document and turns out to be wrong, the argument will be about content, not technology. With no content owner it lands on the developer, which is a dead end.
- 12
You know how often documents change: yearly, monthly or daily.
That decides whether you need incremental re-indexing and freshness monitoring, or whether rebuilding the index on a schedule by hand is enough.
Verifiability: how you will know it worked
- 13
You can write down twenty real questions people actually ask.
Not invented ones — the ones that came into support or were asked in meetings. This is the actual specification: without them quality cannot be measured and the argument about it becomes a matter of taste.
- 14
For each question you know which document holds the correct answer.
It turns “I like it or I don't” into something measurable: did the system find the right document and did it quote that one. Such a set takes a day to assemble and works for years.
- 15
There is an agreement on what counts as an acceptable answer.
“Accurate and complete” is not a criterion. A criterion sounds like: the right document in the top three results, an answer with a source link, and an honest “not found” instead of an invention.
Twelve or more yeses
The data is ready. Start with a pilot on one section of the corpus and your twenty questions: two or three weeks is enough to see whether retrieval lands and whether to expand.
Seven to eleven
The normal state of a working company. You can start, but on the part of the corpus with the most yeses. In parallel, close the gaps in permissions and versioning — those are the most expensive to fix later.
Fewer than seven
RAG right now will give you a beautiful demo and a system that does not work. It is cheaper to spend two or three weeks putting the documents in order: assign an owner, separate the versions, mark what is current. That work is not about AI, and without it any contractor lands in the same place.
Done the checklist? We will review your case
Tell us how it went: how many documents, which answers were no, and what questions the system must close. You get a review of your corpus in return — what to do first and whether you need RAG at all. It is a review, not a sales pitch; if the task has a simpler answer, we will say so.
What does RAG readiness mean?
It is the set of properties that let retrieval produce a verifiable answer: the source is known and singular, the structure allows meaningful chunking, permissions are described, current revisions are distinguishable from superseded ones, and there is a set of questions with known correct answers. The model does not change this — it works with whatever retrieval found.
Do I need to give an email to download the checklist?
No. The file downloads on a button press and asks for nothing. The whole list is also printed on this page — the PDF just makes it easier to take into a meeting.
How long does the checklist take?
About an hour, provided you have someone who knows where the documents live and someone accountable for their content. Versioning and permissions usually take the longest to discuss — which are exactly the two items that cost the most to fix later.
We failed the checklist. What now?
That is a normal and useful outcome: it saves the budget of a pilot that would not have flown anyway. The order of work is always the same — pick one section of the corpus, put it in order, run the pilot there. Expanding from a small working system is easier than from a large broken one.
RAG & knowledge base
What we actually do once the data is ready.
What RAG actually is and when a business needs it
RAG means the model answers from your documents rather than from the internet, with a visible source under every claim. What it is made of, where quality is lost in chunking and updates, how to measure that it works and when you do not need it.
Chunking documents for RAG: why chunks matter more than the model
A bad RAG answer almost always comes from chunking rather than from the model. How to split policies, contracts, scanned PDFs and support tickets, what breaks on a corpus of tens of thousands of long documents, and how to tell that chunking is the culprit.
How to update a RAG knowledge base without stopping the system
A demo lives on a static corpus, production on a moving one. Incremental updates instead of full reindexing, swapping the index without a restart, deleting documents, and which metrics reveal retrieval degradation before users report it.