RAG chunking strategies that actually improve answers

When a retrieval-augmented system gives a wrong or vague answer, people usually blame the model and reach for a bigger one. Nine times out of ten the real problem sits earlier: the right text was never retrieved, because it was split badly. Good RAG chunking strategies decide what the model is even allowed to see.

These are the approaches I use when building RAG systems for clients, in roughly the order I reach for them.

Why chunking matters more than the model

Retrieval works on chunks, not documents. Each chunk becomes one embedding, and a search returns the chunks whose embeddings sit closest to the question. Two failure modes follow from that:

• Chunks too big — one embedding has to represent five different ideas, so it matches none of them strongly. The answer is buried in noise.
• Chunks too small — the sentence that matches the question is retrieved, but the sentence that explains it lives in the next chunk and never arrives.

Every chunking strategy is a way of steering between those two.

Strategy 1: fixed-size chunks with overlap

Split every N tokens, repeating the last few tokens of each chunk at the start of the next. It is the default in most tutorials because it is simple and it works on any text.

My starting point is around 400–600 tokens with 10–15% overlap. It is a baseline, not an answer: fixed-size splitting happily cuts a table in half or separates a heading from the paragraph it introduces. Use it for unstructured text — transcripts, emails, scraped pages — and to get something running on day one.

Strategy 2: structure-aware chunking

Most business documents already tell you where the boundaries are: headings, sections, list items, table rows, FAQ entries. Splitting along that structure gives chunks that each hold one complete idea.

• Markdown and HTML — split on h2/h3, keep the heading path (“Returns › International orders”) attached to every chunk.
• PDF manuals and contracts — split on numbered clauses and sections, not pages.
• FAQs and knowledge bases — one question and its answer per chunk, never split.
• Tables — keep the header row with every group of rows, or convert each row into a sentence.

If a section is still too long, fall back to fixed-size splitting inside that section only. This hybrid is what I ship most often.

Strategy 3: semantic chunking

Semantic chunking embeds each sentence and starts a new chunk where the meaning shifts — where the similarity between neighbouring sentences drops. It helps on long, loosely structured text like interviews or reports that wander between topics.

It costs more to index and is harder to debug, so I only use it when structure-aware splitting clearly fails. It is rarely the first thing to try.

Strategy 4: small chunks for search, big chunks for the answer

This is the one that fixes the most real problems. You index small, precise chunks — a paragraph or a few sentences — because they match questions sharply. But when one is retrieved, you pass its parent section to the model instead.

You get precise retrieval and enough context to answer properly. The implementation is simple: store a parent_id on each small chunk and swap it for the parent before building the prompt, de-duplicating when several children share a parent.

Metadata is part of the chunk

A chunk without context is a paragraph floating in space. For every chunk I store:

• Document title and heading path, prepended to the text before embedding
• Source URL or file, for citations
• Tenant or user id, for permissions
• Language, date and document type, for filtering

Filtering on metadata before the vector search — “only this client’s documents, only the current policy version” — often does more for answer quality than any change to chunk size. It is also how you keep one customer’s data out of another’s answers, which I cover in RAG inside a Laravel app.

Practical defaults

1. Start structure-aware, with fixed-size fallback for long sections.
2. Aim for 300–600 tokens per chunk, 10–15% overlap.
3. Prepend title and heading path to every chunk before embedding.
4. Add parent-child retrieval as soon as answers feel “almost right but missing context.”
5. Filter by tenant, language and version before searching.

Measure before you tune

Do not change chunk size on a hunch. Build a small set of real questions with the passage that should answer each one, and check whether that passage shows up in the top results. Change one thing, run it again, compare. I explain that process in how to evaluate a RAG system.

If your assistant keeps answering from the wrong page of your documentation, send me a few examples — the fix is usually in the chunking, and usually faster than you expect.