Chunk Size and Overlap in a RAG Knowledge Base: Effects on Embedding Count, Token Budget and Re-Indexing

Chunk size and overlap affect how much context a retrieval system can find, how many embeddings it stores and how much text reaches the answering model. The settings are testable choices, not universal constants. A useful decision combines a clear counting convention with representative questions and known supporting passages.

A RAG system retrieves relevant source passages and supplies them to a language model as material for an answer. A chunk is a document fragment. Overlap means that neighbouring chunks repeat part of the same text. An embedding is a numerical representation used for similarity search; a RAG system can also combine vector retrieval with keyword search. Tokens are units of the selected tokenizer, not fixed fractions of words. Language, punctuation and model all affect the count.

An exact count for a simple splitter

The following model uses one already-tokenized document, fixed windows and a shortened final chunk without padding. It stops as soon as the last token is covered. Let N be document length, C chunk size and O overlap, with 0 ≤ O < C. The step is C − O. Empty documents produce no chunks; a document no longer than C produces one.

step = C - O
chunks = 1 + ceil(max(0, N - C) / step)   # for N > 0
total token occurrences = N + O * (chunks - 1)

The function ceil rounds upward to the next integer.

For N = 200,000 and C = 512, the result is:

Multiplying the chunk count by 512 would count nonexistent padding in the last chunk. For long documents the expansion approaches C / (C − O); finite documents still have boundary effects. At 50% overlap the interior is mostly repeated, but not every token appears exactly twice. Sentence-aware splitting, per-chunk retokenization and extra headings can change these totals.

Text repetition is not the same as vector storage or a bill

The table counts repeated text tokens. With one fixed-dimensional vector per chunk, the vector count is 447, 521 or 781. The dimension is the number of values in a vector. Vector bytes depend on that dimension and the storage format of each number; the search index, stored text, metadata and replicas add separate overhead. A 14% text increase is not automatically a 14% total storage or price increase.

Embedding services may charge by tokens, characters or another unit, while self-hosted models incur compute costs. Repeated ingestion, model changes and document updates also cost work. The actual provider, model, caching policy and stored representation determine the bill; embedding the corpus is not necessarily cheap or a one-time operation.

Overlap preserves context, but has no universal optimum

Overlap can preserve relationships across boundaries, such as a definition followed by an exception. It does not guarantee that every sentence or answer fits whole into a chunk. Long tables, code, references and multi-paragraph explanations need their own treatment. Two sentences do not have a fixed token length.

Splitting at headings and paragraphs can improve coherence, followed by a size limit for oversized sections. A heading path such as Handbook > Billing > Cancellation can restore useful context. The added text still occupies input and prompt space, and its benefit needs evaluation. Structure-aware splitting is a promising strategy, not proof that overlap is unnecessary.

A compact evaluation can compare several sizes and overlaps using the same questions, source versions and retrieval settings. Relevant measures include whether the required supporting passages are retrieved, answer correctness, citation support, latency and total cost. Small chunks can lose context; large chunks can include distracting topics. No 400–800-token range is exempt from those trade-offs. An embedding is learned, not a literal arithmetic average of its topics.

The size of the assembled model input

Eight full chunks of 512 tokens contain 4,096 tokens of source text under the same tokenizer assumption. Instructions, question, conversation history, headings and citations add to that amount; the answer needs its own output budget. The actual prompt tokenizer and model accounting must be checked after assembly.

Halving chunk size and doubling the number of hits preserves only that nominal source-text total. It does not guarantee better retrieval, identical latency or an unchanged bill. Search, query embeddings and reranking can have separate costs. Retrieval chunks and generation context also need not be identical: a parent-document retriever can search small fragments and return a larger surrounding section.

Adjacent hits may repeat overlapping text. Merging verified source spans or removing exact duplicates can save context space. The source version and text positions must agree, so that only passages from the same document version are merged. Automatically discarding every lower-scoring neighbour can also discard the only passage containing an exception, so diversity and deduplication rules need the same quality checks as chunk size.

Re-indexing changed inputs with traceable versions

A different split changes the embedding input for affected chunks. It does not make incremental migration impossible or require every document to be processed at once. Unchanged exact inputs can reuse compatible embeddings; a versioned index can be built and tested before replacing the active version. Old references need an explicit retention or redirection policy rather than accidental invalidation.

Useful records include source ID and version, source offsets, splitter version, complete embedding input and embedding configuration. The source association can be a metadata field or mapping; it need not be encoded in the chunk ID itself. A text checksum alone is insufficient for reuse if model version, dimensions, task type, preprocessing or prepended context changed.

Raw documents, extracted text, chunks and embeddings should remain distinguishable stages. Changing chunking reruns the splitter before embedding changed inputs; changing only the embedding model can reuse prepared chunks if the new limits and input requirements permit this. Deleted source passages also need removal from the active index.