How to Choose the Right Chunking Strategy for RAG Systems

The right RAG chunking strategy starts with the questions users ask and the structure of the source material. Preserve headings, paragraphs, lists, tables, page references, and document identity during parsing. Create retrieval units that each express a coherent idea. Keep a link from every unit to a larger parent section so the system can recover surrounding context after it finds a precise match.

Begin with a simple baseline: split on structural boundaries, enforce a maximum token size, add little or no overlap, and retrieve parent context when needed. Then evaluate at least two alternatives on real questions. Use smaller units when queries target specific facts or clauses. Use larger units when answers depend on narrative, chronology, several adjacent facts, or the meaning of a whole section. Use hierarchical or multi-level retrieval when the same corpus must support both.

There is no research-backed number that works for every corpus. A 300-token chunk may be excellent for a policy clause and destructive for a table with its explanatory notes. Chunk size is one variable among parsing, boundaries, metadata, retrieval, and the question.

The original retrieval-augmented generation paper paired a generator with passages selected from a dense Wikipedia index. That architecture made the retrieval unit a first-class design choice: the retriever does not search an abstract document collection; it ranks whatever units you created during indexing.

A unit that is too large can bury the relevant sentence inside unrelated text. Its embedding must compress several topics into one vector, and the generator pays to read all of them. A unit that is too small can lose the subject, section purpose, exception, or definition needed to interpret it. It may retrieve well by lexical similarity yet fail as evidence.

Write down the unit your answer needs before choosing a splitter. If users ask, ‘What notice period applies to this cancellation?’, a self-contained clause may be enough. If they ask, ‘Why did the underwriting decision change between submission and renewal?’, the useful unit may span a timeline, emails, endorsements, and a later decision. The second problem is not solved by increasing overlap around arbitrary token windows. It needs connected context.

A chunker cannot repair a bad parse. Before splitting, determine whether the pipeline retained headings, reading order, page numbers, list nesting, table cells, captions, footnotes, and document versions. A PDF converted into a flat stream may join two columns, detach a table header, or place a footnote beside the wrong paragraph.

Use natural boundaries first: title, section, subsection, paragraph, list item, table, or transcript turn. Combine adjacent blocks until they approach the working size limit, but avoid crossing a strong semantic boundary merely to fill the quota. Store breadcrumb metadata such as document title, section path, page range, effective date, and access scope with each unit.

Do not force every content type through the same rule. Keep a table with its header and units. Attach a figure caption to the figure reference. Preserve a contract definition wherever a downstream clause depends on it, either through parent retrieval or an explicit link. For transcripts, speaker and timestamp may matter more than paragraph length.

This is also where provenance begins. A generated answer should be able to point to the original page or passage, not only an opaque vector-store identifier. If a chunk cannot be inspected in its source setting, it is a weak foundation for reviewable knowledge work.

The Dense X Retrieval study compared passages, sentences, and generated propositions across five open-domain question-answering datasets. A proposition was designed as one minimal, self-contained fact. Proposition-level indexing improved retrieval and downstream QA on average in that experimental setting, especially for unsupervised dense retrievers.

The useful principle is information density, not ‘always split into propositions’. A compact, self-contained unit gives the retriever less irrelevant material to confuse with the query. But converting documents into propositions adds an extraction model, far more index entries, and another place to introduce unsupported or context-poor text. The paper itself evaluated generated propositions for faithfulness and whether they stood alone. A production system should retain the original passage beside any transformed unit.

A practical compromise is child-to-parent retrieval. Index a paragraph, sentence, clause, or proposition as the searchable child. Once it matches, return the containing section or a controlled neighbourhood to the generator. The child improves precision; the parent restores definitions, exceptions, and nearby reasoning. This is often safer than making every indexed unit large enough for every possible question.

Do not assume fine-grained units will transfer from Wikipedia QA to legal, scientific, or operational documents. Test the corpus you have. In structured work, a numbered clause or evidence claim may be a stable child unit. In narrative reports, the paragraph may be the smallest unit that still makes sense.

Some questions are harmed by aggressive fragmentation. LongRAG tested much longer retrieval units with long-context readers. On its Wikipedia experiments, related documents were grouped into roughly 4,000-token units; on two long-document datasets, individual documents were retrieved as units. The approach performed strongly on four QA tasks while retrieving only a few long units.

That does not mean every system should index whole documents. Long units shift work from the retriever to the reader. They reduce the number of candidates and preserve semantic continuity, but they increase the amount read after a match and may return too much material for precise citations. The right trade-off depends on document length, model capability, latency, and whether the question asks for a fact or a whole-document view.

RAPTOR offers another pattern: recursively cluster text and create summaries, forming a tree that can retrieve detailed leaves or higher-level themes. This is useful when one corpus must answer both ‘What does this paragraph say?’ and ‘What themes recur across the report?’ Generated summaries are derived evidence, not the source itself. Keep their child links and let reviewers descend to the original text.

Long and hierarchical retrieval are most useful when the question genuinely needs synthesis. They are extra machinery for simple lookup. Add them after a flat structural baseline shows a repeatable failure on broad, multi-hop, or thematic questions.

Overlap is a blunt response to boundary loss. It can preserve a sentence cut between windows, but it also duplicates storage, crowds top-k results with near-identical text, and makes citations ambiguous. If structural splitting already respects paragraphs and sections, a fixed percentage of overlap may add little.

The Late Chunking paper tested a different way to preserve context. Instead of embedding each chunk independently, its method lets a long-context embedding model process the larger text first, then pools token representations into chunk embeddings. The resulting units remain small for retrieval while their vectors contain information from the surrounding document.

Late chunking still needs chunk boundaries and a compatible long-context embedding model. It also does not make the retrieved text self-explanatory; the generator may still need parent context. In the paper's fixed-size experiments, a 16-token overlap did not show a clear advantage across the tested BEIR tasks. That is evidence against treating overlap as free accuracy, not proof that overlap never helps.

Test no overlap against a modest boundary overlap where sentences or OCR blocks are frequently split. Measure duplicate retrieval, unique evidence coverage, index growth, and answer quality. If overlap only produces three versions of the same paragraph in the prompt, remove it.

Build a test set that reflects the corpus: direct facts, definitions, exceptions, tables, comparisons, multi-section reasoning, older versions, and unanswerable questions. For each item, mark the source span needed for a good answer. Run every chunking candidate with the same parser, embedding model, retriever, reranker, and context budget so the comparison isolates the unit design.

At retrieval time, measure whether the required evidence appears in the first k results, how much irrelevant text accompanies it, and how many results are duplicates from one source region. At answer time, check faithfulness, completeness, citation precision, latency, and context tokens. Review failures by tracing the question to retrieved units and back to their parent document.

Include slices. A strategy can win overall while failing on tables or policy exceptions. Compare small structural chunks, the same chunks with parent expansion, and a larger section-level baseline. Add propositions, hierarchical summaries, late chunking, or long units only when the simpler systems expose a specific weakness they are designed to address.

A longer context window does not remove the need for selection. The peer-reviewed Lost in the Middle study found that model performance could vary with the position of relevant information, with evidence in the middle of long contexts often used less reliably. Context capacity and context use are different properties.

The best chunking strategy is therefore not the one with the most sophisticated splitter. It is the smallest system that repeatedly retrieves enough meaning for the real question, preserves a trace to the source, and remains understandable when it fails. If your knowledge workflow keeps finding the right words without the context needed to act on them, share the document type, question, and broken evidence path at sahil@granveo.com.