Chunking documents for RAG: why chunks matter more than the model
A bad RAG answer almost always comes from chunking rather than from the model. How to split policies, contracts, scanned PDFs and support tickets, what breaks on a corpus of tens of thousands of long documents, and how to tell that chunking is the culprit.
9 min read
Answer quality is decided before the model
In RAG the model answers only from the fragments handed to it. If the needed paragraph never made the retrieval set, no model swap fixes that: the model does not know something is missing and answers confidently from what it has.
So diagnosis starts not with the prompt but with the question “was the right fragment in the retrieval set at all”. In most projects we have looked at, the answer to that question was the diagnosis.
What a long document actually breaks
A corpus of tens of thousands of long documents breaks two things at once. First, a single vector for a long text is useless: the averaged meaning of a hundred-page policy resembles no specific question. Second, the smaller the chunks the more of them there are, and the more often fragments that read alike but mean different things crowd each other out of the top results.
Hence a practical rule: split along the document's own semantic boundaries rather than by character count, and keep a chunk large enough to answer one question completely. Overlap between neighbouring chunks exists precisely so a thought is not torn in half — not as a general precaution.
Four chunking strategies that genuinely differ
The difference between them is not fashion but what becomes the unit of meaning.
- By document structure — headings, clauses, contract articles. The best option for policies and regulations: the author already placed the boundaries.
- Sliding window with overlap — when there is no structure: transcripts, correspondence, old PDFs without markup.
- Parent and child — retrieve on small chunks, hand the enclosing section to the model. This resolves the conflict between retrieving well on small units and answering well on large ones.
- Tables chunked separately — a row together with its column headers. A table split as if it were prose loses its meaning entirely while still being retrieved confidently.
Different sources, different rules
One chunking rule for the whole corpus is the most common reason a system “mostly works but misses on half the questions”. The source type dictates both the boundaries and what has to sit alongside the text.
- Policies and contracts: boundaries by clause, with version and effective date in the metadata — otherwise the system answers from a superseded revision.
- Support tickets: question and resolution in one chunk. Split per message they become a set of remarks with no answer.
- Scans and PDFs: recognition and column-order reconstruction first, chunking only after. Garbage at the input is cured neither by chunk size nor by the model.
- Wikis and product knowledge bases: the page as the unit, the section heading as a mandatory chunk prefix.
Metadata beats another embedding
Each chunk should carry its source, section, version, date and access rights. That gives three things at once: filtering before the vector search, an honest citation in the answer, and the ability not to show an employee what they are not entitled to see.
A metadata filter usually buys more precision than moving to a pricier embedding model, and costs incomparably less to run.
How to tell chunking is the problem
You need a small set of real questions annotated with the document that holds the answer. Then you check one thing: does the right fragment make the top of the retrieval set. If it does and the answer is still poor, the problem is the prompt or the model. If it does not, chunking or the index is at fault and swapping models is pointless.
A set of thirty to fifty questions takes a day to assemble and pays for itself at every pipeline change. Without it, any chunking edit is an argument between two opinions.