{"id":10410,"date":"2026-09-15T07:20:00","date_gmt":"2026-09-15T05:20:00","guid":{"rendered":"https:\/\/www.lukaswojcik.com\/blog\/?p=10410"},"modified":"2026-10-26T13:14:14","modified_gmt":"2026-10-26T12:14:14","slug":"splitting-documents-for-ai-search-how-section-size-and-overlap-affect-results","status":"publish","type":"post","link":"https:\/\/www.lukaswojcik.com\/blog\/en\/cloud-ai\/splitting-documents-for-ai-search-how-section-size-and-overlap-affect-results\/","title":{"rendered":"Splitting Documents for AI Search: How Section Size and Overlap Affect Results"},"content":{"rendered":"

Chunk size and overlap affect how much context a retrieval system can find, how many embeddings it stores and how much text reaches the answering model. The settings are testable choices, not universal constants. A useful decision combines a clear counting convention with representative questions and known supporting passages.<\/p>\n

A RAG system retrieves relevant source passages and supplies them to a language model as material for an answer. A chunk is a document fragment. Overlap means that neighbouring chunks repeat part of the same text. An embedding is a numerical representation used for similarity search; a RAG system can also combine vector retrieval with keyword search. Tokens are units of the selected tokenizer, not fixed fractions of words. Language, punctuation and model all affect the count.<\/p>\n

\"Fixed-window
Fixed-window example without padding: increasing overlap from 64 to 128 tokens increases the count from 447 to 521 chunks. Text repetition, vector storage and model billing remain separate quantities.<\/figcaption><\/figure>\n

An exact count for a simple splitter<\/h2>\n

The following model uses one already-tokenized document, fixed windows and a shortened final chunk without padding. It stops as soon as the last token is covered. Let N be document length, C chunk size and O overlap, with 0 \u2264 O < C. The step is C \u2212 O. Empty documents produce no chunks; a document no longer than C produces one.<\/p>\n

step = C - O\nchunks = 1 + ceil(max(0, N - C) \/ step)   # for N > 0\ntotal token occurrences = N + O * (chunks - 1)<\/code><\/pre>\n
The function ceil rounds upward to the next integer.<\/p>\n
For N = 200,000 and C = 512, the result is:<\/p>\n\n\n\n\n\n\n\n
Overlap<\/th>\nStep<\/th>\nChunks<\/th>\nToken occurrences<\/th>\nExtra text<\/th>\n<\/tr>\n<\/thead>\n
64 (12.5%)<\/td>\n448<\/td>\n447<\/td>\n228,544<\/td>\n14.272%<\/td>\n<\/tr>\n
128 (25%)<\/td>\n384<\/td>\n521<\/td>\n266,560<\/td>\n33.280%<\/td>\n<\/tr>\n
256 (50%)<\/td>\n256<\/td>\n781<\/td>\n399,680<\/td>\n99.840%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Multiplying the chunk count by 512 would count nonexistent padding in the last chunk. For long documents the expansion approaches C \/ (C \u2212 O); finite documents still have boundary effects. At 50% overlap the interior is mostly repeated, but not every token appears exactly twice. Sentence-aware splitting, per-chunk retokenization and extra headings can change these totals.<\/p>\n
Text repetition is not the same as vector storage or a bill<\/h2>\n
The table counts repeated text tokens. With one fixed-dimensional vector per chunk, the vector count is 447, 521 or 781. The dimension is the number of values in a vector. Vector bytes depend on that dimension and the storage format of each number; the search index, stored text, metadata and replicas add separate overhead. A 14% text increase is not automatically a 14% total storage or price increase.<\/p>\n
Embedding services may charge by tokens, characters or another unit, while self-hosted models incur compute costs. Repeated ingestion, model changes and document updates also cost work. The actual provider, model, caching policy and stored representation determine the bill; embedding the corpus is not necessarily cheap or a one-time operation.<\/p>\n
Overlap preserves context, but has no universal optimum<\/h2>\n
Overlap can preserve relationships across boundaries, such as a definition followed by an exception. It does not guarantee that every sentence or answer fits whole into a chunk. Long tables, code, references and multi-paragraph explanations need their own treatment. Two sentences do not have a fixed token length.<\/p>\n
Splitting at headings and paragraphs can improve coherence, followed by a size limit for oversized sections. A heading path such as Handbook > Billing > Cancellation<\/em> can restore useful context. The added text still occupies input and prompt space, and its benefit needs evaluation. Structure-aware splitting is a promising strategy, not proof that overlap is unnecessary.<\/p>\n
A compact evaluation can compare several sizes and overlaps using the same questions, source versions and retrieval settings. Relevant measures include whether the required supporting passages are retrieved, answer correctness, citation support, latency and total cost. Small chunks can lose context; large chunks can include distracting topics. No 400\u2013800-token range is exempt from those trade-offs. An embedding is learned, not a literal arithmetic average of its topics.<\/p>\n
The size of the assembled model input<\/h2>\n
Eight full chunks of 512 tokens contain 4,096 tokens of source text under the same tokenizer assumption. Instructions, question, conversation history, headings and citations add to that amount; the answer needs its own output budget. The actual prompt tokenizer and model accounting must be checked after assembly.<\/p>\n
Halving chunk size and doubling the number of hits preserves only that nominal source-text total. It does not guarantee better retrieval, identical latency or an unchanged bill. Search, query embeddings and reranking can have separate costs. Retrieval chunks and generation context also need not be identical: a parent-document retriever can search small fragments and return a larger surrounding section.<\/p>\n
Adjacent hits may repeat overlapping text. Merging verified source spans or removing exact duplicates can save context space. The source version and text positions must agree, so that only passages from the same document version are merged. Automatically discarding every lower-scoring neighbour can also discard the only passage containing an exception, so diversity and deduplication rules need the same quality checks as chunk size.<\/p>\n
<\/p>\n
\"The
The cello lies in its open case, and a pocket stays free for bow, rosin and cloth. Likewise, the input to an AI model needs room for everything else besides the text pieces.<\/figcaption><\/figure>\n
Re-indexing changed inputs with traceable versions<\/h2>\n
A different split changes the embedding input for affected chunks. It does not make incremental migration impossible or require every document to be processed at once. Unchanged exact inputs can reuse compatible embeddings; a versioned index can be built and tested before replacing the active version. Old references need an explicit retention or redirection policy rather than accidental invalidation.<\/p>\n
Useful records include source ID and version, source offsets, splitter version, complete embedding input and embedding configuration. The source association can be a metadata field or mapping; it need not be encoded in the chunk ID itself. A text checksum alone is insufficient for reuse if model version, dimensions, task type, preprocessing or prepended context changed.<\/p>\n
Raw documents, extracted text, chunks and embeddings should remain distinguishable stages. Changing chunking reruns the splitter before embedding changed inputs; changing only the embedding model can reuse prepared chunks if the new limits and input requirements permit this. Deleted source passages also need removal from the active index.<\/p>\n
\n
Questions and answers<\/h2>\n
Does structure-aware splitting keep all other chunks unchanged?<\/h3>\n
Not necessarily. Section packing, size limits and changed heading paths can affect later chunks. Fixed windows can shift many later boundaries after an insertion, but even that is not an unconditional statement about every later input. Comparing actual complete inputs is more reliable than assuming stability from the splitter name.<\/p>\n
What happens when an embedding input is too long?<\/h3>\n
The model and endpoint determine the limit and behavior. Google’s documented prediction endpoint for gemini-embedding-001<\/code> supports autoTruncate<\/code>, defaulting to true; false makes an overlong text fail. Returned statistics include truncated<\/code> and token_count<\/code>. This must not be generalized to every embedding API. Headings and other embedded additions count too.<\/p>\n
A discarded tail does not contribute to that vector, reducing retrieval evidence for its content. The chunk can still be retrieved through its retained beginning or another search method; absolute unfindability does not follow. Logging truncation and checking actual model-token counts prevents a long stored passage from creating a false impression that all of it was embedded.<\/p>\n
Is a single-document update enough?<\/h3>\n
Usually the affected source and its derived records can be updated independently, provided source tracking and cleanup are implemented. Unchanged inputs can be reused with a compatible embedding configuration. A safe update also handles removed chunks and avoids mixing incompatible index versions during retrieval.<\/p>\n<\/div>\n
\n
Sources<\/h2>\n