In one sentence
A chunk is a small piece of a document, created so a retrieval system can search and return it usefully.
The simple model
A RAG system does not search whole documents. A fifty-page handbook is too large to place in the model's context and too coarse to match against a specific question. So documents are split into smaller pieces, called chunks, before they are made searchable.
When a question comes in, the system finds the most relevant chunks and passes those to the model, rather than the entire document.
Why you care
How documents are chunked has a large effect on whether retrieval actually works, and it is one of the most common reasons a RAG system gives poor answers.
- Chunks that are too large bring in a lot of irrelevant text with the relevant part.
- Chunks that are too small lose the surrounding context needed to make sense, like a sentence cut off from the paragraph that explains it.
- Chunks split in the wrong place, mid-table or mid-list, can become meaningless.
Good chunking usually follows the natural structure of the document: sections, headings, list items. This is why preparing documents for AI matters so much.
A small example
A policy assistant kept giving half-answers. The cause was chunking that split policies in the middle, so the condition landed in one chunk and the exception in another. Re-chunking along the document's own headings, one policy per chunk, fixed most of the wrong answers without changing anything else.
What this does not solve
Good chunking helps the right passage be found; it does not make the passage correct, current or well interpreted. It is one ingredient of reliable retrieval, not the whole recipe.
