Chunk Size and Overlap in a RAG Knowledge Base: Effects on Embedding Count, Token Budget and Re-Indexing
Contents
A knowledge base for an AI assistant begins by cutting long documents into short chunks. Two numbers decide how that happens: how long a chunk should be, and how far two consecutive chunks overlap. Both are entered in a minute, apply to the whole corpus, and can only be changed afterwards by running everything through again from the start.
And both are paid twice: once when the knowledge base is built, and then again at every single question. Which is why they are worth calculating rather than guessing.
The unit of measurement is the token – roughly a word fragment; a hundred tokens is about seventy-five words of English prose. Every finished chunk is translated into a long sequence of numbers that captures its meaning. That sequence is called an embedding, and the search later runs through those numbers rather than through the text.
Why the Overlap Costs Money
Chunks do not advance by their full length, only by the part that does not overlap. With a chunk size of 512 tokens and an overlap of 64, each chunk advances by 448. That means more chunks for the same document – and because every chunk is stored at full length, the overlapping part ends up in storage twice.
The table shows what that means for a document of 200,000 tokens at a chunk size of 512.
| Overlap | Chunk advances by | Number of chunks | Stored in total | Overhead |
|---|---|---|---|---|
| 64 tokens (12 %) | 448 | 447 | 228,864 | +14 % |
| 128 tokens (25 %) | 384 | 521 | 266,752 | +33 % |
| 256 tokens (50 %) | 256 | 781 | 399,872 | +100 % |
The last row is the one worth remembering: an overlap of half the chunk size stores every token twice. The overhead does not grow evenly with the overlap but shoots up as soon as the advance gets narrow. That is why the useful range sits between ten and twenty per cent, and why anything above a third needs a reason.
What the Overlap Is Actually For
A fixed split cuts wherever the counter runs out, which is regularly in the middle of the sentence that contains the answer. The overlap exists so that the sentence appears whole in at least one chunk – and that is its entire job.
Which means the overlap only has to be as large as the passage that must not be torn apart. For prose, two or three sentences is generous, and two or three sentences is about sixty to a hundred tokens. Numbers larger than that are usually chosen out of caution and paid for in storage without buying anything: a sentence that already fits whole into one chunk does not fit more whole because it also appears in the next.
Splitting at Structure Beats Splitting at a Number
The better fix for the same problem is not to cut mid-sentence in the first place. Splitting at headings, then at paragraphs, and only falling back to a token count when a paragraph exceeds the size, removes most of the reason the overlap exists – and it produces chunks whose contents belong together, which is what the embedding is supposed to represent.
The addition that costs almost nothing is to store the heading path with each chunk and prepend it to the text. A chunk that begins with Handbook > Billing > Cancellation retrieves better and reads better when it arrives in the prompt, because it carries the context that the split took away.
Two Sizes, Two Different Failures
| Chunk size | What it does well | How it fails |
|---|---|---|
| 128 – 256 | precise retrieval, the hit is exactly the passage | the passage arrives without the sentence that explains it |
| 400 – 800 | a paragraph with its context, one topic per embedding | barely at all; this is the working range |
| 1024 – 2048 | whole chapters, nothing is ever cut off | the embedding averages several topics and matches nothing sharply |
The failure of large chunks is the one that is hard to see from the outside. An embedding is a single point for the whole text, so a chunk covering three subjects sits somewhere between all three and is close to none of them. Retrieval does not report this; it simply returns something else instead, and the answer is built from a passage that was not quite the right one.
The Cost That Recurs
Embedding a corpus is cheap and happens once. But the chunk size is also the unit in which the retrieved text is measured, the text the assistant is handed at every question – and that is paid for every time. Eight chunks of 512 tokens is 4096 tokens the model reads before it has even seen the question.
That is the number to size against, because it multiplies by the number of questions rather than by the size of the corpus. Halving the chunk size while doubling the number of retrieved chunks keeps the bill identical and the precision higher; halving it without touching the count halves the bill and risks arriving with too little context. Both are decisions, and both sit in the same line.
Changing the Setting Means Rebuilding
The chunk boundaries are baked into the embeddings. A different size produces different texts, different embeddings and different identifiers, so there is no partial migration – the corpus is embedded again, and any identifiers stored elsewhere point at chunks that no longer exist.
Which argues for one habit at the start: keeping the raw documents, the split and the embedding as three separate steps with the intermediate result written to disk. Re-embedding is then a rerun of the last step on a corpus that is already prepared, rather than an afternoon of reassembling how the text got in there in the first place.
Sources
Related articles
