Tutorial: Testing Chunk Size and Overlap Against a Retrieval Test Set

A chunk is a section of a source document stored as a retrieval unit. Overlap means that neighboring chunks repeat some of the same source text. Chunk size affects how much useful evidence and unrelated text reach the answer model. There is no universally best size or overlap: a comparison needs fixed source material, checked questions and a clearly defined success criterion.

This tutorial evaluates retrieved evidence without generating answers. Embedding models, indexing and search still require compute and may incur charges. The Python example runs locally on synthetic ranked spans; it is an evaluator, not a complete embedding pipeline.

Fixed sources and labeled evidence

Each question needs an ID, its wording, an immutable document-version ID and a verified evidence span. The span uses Python character offsets: start inclusive, end exclusive. The quoted text must equal that exact slice of the frozen source. Bytes, UTF-16 positions and token positions are different coordinate systems.

Offsets are established after extraction and normalization, before chunking. Every chunk retains its document-version ID and source interval. Searching for the quote alone can accept the same sentence in the wrong document or the wrong occurrence. Any preprocessing that changes the text requires a reliable mapping back to these coordinates.

Synthetic questions can help populate a first test set, but generating from a whole document does not remove source-wording bias. Human review checks answerability, evidence, paraphrases and task coverage. Real queries add useful cases, subject to privacy and labeling quality. Unanswerable or multi-document questions need separate evaluation rules; the example below covers one contiguous evidence span per answerable question.

A metric chosen before comparing settings

Single-span hit@k is the fraction of questions whose complete labeled span fits inside at least one of the first k chunks. Union hit@k accepts coverage across several retrieved chunks of the same document version, provided no gap remains. The second measure detects split evidence; it does not prove that the answer model can reconstruct the passage correctly.

Single-span MRR@k averages the reciprocal rank of the first chunk containing the whole span, with zero for a miss. A complete match at rank 1 contributes 1, at rank 2 it contributes 0.5, and at rank 4 it contributes 0.25. These are explicit measures for this labeling scheme. In general information retrieval, recall asks what fraction of all relevant units was retrieved. A hit on one relevant unit is not the same as recovering several required evidence passages.

Thirty or fifty questions can be a pilot, not a universal sufficient sample. One changed outcome is 3.33 percentage points for 30 questions and 2 points for 50. Small differences need more evidence, per-question comparisons and an appropriate uncertainty analysis. A held-out set prevents repeated tuning from being mistaken for independent validation.

A small executable evaluator

The standard-library-only file runs with python3 evaluate.py. Rankings are supplied in retrieval order; an empty list means a completed search with no results, while a missing ranking is an error. Duplicate spans, invalid coordinates and mismatched evidence quotes are rejected.

def evaluate(questions, rankings, corpus, k=5):
    # corpus keys identify immutable document versions.
    if type(k) is not int or k < 1 or not questions:
        raise ValueError("positive k and nonempty questions required")
    def valid_span(s):
        doc, start, end = s
        if (doc not in corpus or type(start) is not int
                or type(end) is not int
                or not 0 <= start < end <= len(corpus[doc])):
            raise ValueError("invalid source span")
        return doc, start, end
    hits, union_hits, reciprocal = 0, 0, 0.0
    seen = set()
    for q in questions:
        if q["id"] in seen:
            raise ValueError("duplicate question id")
        seen.add(q["id"])
        doc, start, end = valid_span(q["evidence"])
        if q["quote"] != corpus[doc][start:end]:
            raise ValueError("quote does not match the source span")
        # A missing result is not silently treated as an empty ranking.
        rows = rankings[q["id"]]
        if len({tuple(c) for c in rows}) != len(rows):
            raise ValueError("duplicate retrieved span")
        rows = [valid_span(c) for c in rows]
        top = rows[:k]
        rank = next((n for n, (d, a, b) in enumerate(top, 1)
                     if d == doc and a <= start and b >= end), None)
        if rank is not None:
            hits += 1
            reciprocal += 1 / rank
        # Union coverage: adjacent intervals may cover a split sentence.
        cursor = start
        spans = sorted((max(a, start), min(b, end)) for d, a, b in top
                       if d == doc and a < end and b > start)
        for a, b in spans:
            if a > cursor:
                break
            cursor = max(cursor, b)
        union_hits += cursor >= end
    n = len(questions)
    return {"single_hit_at_k": hits / n,
            "union_hit_at_k": union_hits / n,
            "single_mrr_at_k": reciprocal / n}

if __name__ == "__main__":
    corpus = {"manual-v1": "AA BBBB CC", "other-v1": "AA BBBB CC"}
    questions = [{"id": "q1", "evidence": ["manual-v1", 3, 7],
                  "quote": "BBBB"}]
    rankings = {"q1": [("other-v1", 0, 10),
                       ("manual-v1", 0, 5),
                       ("manual-v1", 5, 10)]}
    print(evaluate(questions, rankings, corpus, k=3))
    # single_hit_at_k: 0.0, union_hit_at_k: 1.0, single_mrr_at_k: 0.0

The synthetic example returns single-span hit 0, union hit 1 and single-span MRR 0. The first result contains the same text in a different document; it does not count. The next two results cover the correct span together. These values test the evaluator and say nothing about embedding quality.

Connecting the evaluator to the actual search

For each configuration, the pipeline splits the same frozen corpus, embeds the chunks, builds an isolated index and retrieves ranked source spans for every question. The adapter must map each returned chunk to its original interval. The code does not supply a tokenizer, splitter, embedding client or search index.

Four candidate sizes such as 256, 512, 1024 and 2048 tokens, each with zero or 20% overlap, give eight configurations. Candidate sizes must fit the chosen embedding model and tokenizer. Silent truncation can make an indexed vector represent less text than the stored chunk; truncation must be disabled where supported or detected and treated as a failed configuration.

The model version, query/document task settings, extraction, similarity function, filters and search parameters stay fixed. Query embeddings can be reused when their complete inputs and model settings are identical. Reuse of document embeddings is safe only for identical embedding inputs. Model changes require a fresh comparison.

Quality, context and cost

At fixed k, larger chunks can deliver more text. The product k × chunk_size is only a nominal maximum for full-sized chunks. Short final chunks, wrappers, deduplication and truncation change actual prompt length. A fixed-budget comparison must count the final serialized context with the answer model’s tokenizer and use the same packing rule for every setting.

For a long fixed-window stream with overlap fraction r, the token multiplier approaches 1 / (1 - r). At r = 0.20 that is 1.25: about 25% extra, not 20%. Four zero-overlap and four 20%-overlap passes over two million tokens therefore approach 18 million document-embedding tokens. Document boundaries and rounding change the exact amount; query embeddings, indexing, storage and evaluation work are additional costs.

A useful results table records configuration, question count, both hit measures, MRR, actual context tokens, embedded tokens, index size and latency. No measured quality curve is supplied here. A recommendation for 512 tokens cannot be established without an identifiable corpus, labels and run output. Structure-aware splitting is another candidate, not an automatic winner.

Separate checks of the full answer pipeline

With reranking, evidence coverage matters both in the initial candidate set and after reranking and context packing. A high first-stage score does not guarantee that the final five chunks retain the evidence. The final answer also needs checks for correct conditions, citations and unsupported claims.

Re-evaluation follows meaningful changes in documents, queries, models or the pipeline. Neither an annual schedule nor a fixed review time guarantees coverage. Versioned questions and per-query results make regressions easier to locate.

Sources

• Sentence Transformers: evaluation metrics
• Chroma: evaluating chunking strategies
• Caspari et al.: multi-objective chunking evaluation
• Google Cloud: text embeddings and truncation