Cut strategies: the catalog

The problem

You cannot hand the retriever the whole document at once. The first reason is purely technical: the embedding model accepts text of limited length, and one giant fragment simply will not fit. The second reason is subtler and more important: even when a long context fits, the model uses information in its middle worse - the "lost in the middle" effect, measured on several models: accuracy is higher when the needed fact stands at the beginning or the end of the context, and noticeably sags in the center (Liu et al., 2023, arXiv:2307.03172). So a large document is cut into chunks - retrieval-sized fragments that are then turned into embeddings, stored in the index, and retrieved one at a time.

The question is not whether to cut but where to cut and how large. This is the chunking stage - one of the stops on the high-level route map. The key idea of the chapter: the choice of cutting strategy is a measurable trade-off between simplicity, cost, and boundary quality, not a single correct constant.

RAG as a method - "find, augment, generate" - was introduced in Lewis et al., 2020, arXiv:2005.11401; chunking is the first step of the retrieval stage: without cutting there is nothing to turn into embeddings and nothing to search.

What chunking consists of

When detailed, the chunking stage breaks into three parts. This is the general frame of any strategy - only the splitter changes.

• Splitter - decides WHERE to cut. On this depends whether related meaning lands in one chunk or a cut tears a sentence in half. The whole catalog below is different splitters.
• Overlap - the shared tail of text that repeats at the end of one chunk and the start of the next. Without overlap, a fact that landed on a cut boundary will not be found in full in any one chunk; a small overlap preserves context at the seam (Pinecone, Chunking strategies).
• Metadata - the source, position (fromChar/toChar), and tags attached to the chunk. Later they are needed to filter candidates in the vector database and to return an exact citation in the generation answer.

Chunk size and overlap are a trade-off, not a constant: small chunks hit the query more precisely but lose surrounding context; large ones carry context but blur relevance and run into "lost in the middle" (Liu et al., 2023, arXiv:2307.03172). The right value depends on your documents, so it is chosen by measuring on a golden set of questions, not by guessing (Pinecone, Chunking strategies).

What to measure size in: characters or tokens

Chunk size can be counted in characters or in tokens, and this is not a cosmetic difference. The character count is simple: a string's length in Python is len(text), no dependencies. But the real budget that RAG runs into is measured not in characters but in tokens: both the model's context window and the bill for a call are counted in tokens, not characters. One token is on average a piece of a word, and the character-to-token ratio differs for different languages and alphabets (Cyrillic usually yields more tokens for the same text than Latin). So a chunk of 1000 characters may unexpectedly fail to fit into the token budget of the context-assembly step.

That is why production splitters count tokens specifically, with the same tokenizer as the target model. OpenAI publishes tiktoken - the tokenizer of its models (OpenAI, tiktoken; OpenAI Embeddings guide); LangChain wraps it in TokenTextSplitter, which cuts by tokens rather than characters (LangChain, split by token). Any strategy from the catalog below can be implemented as character-based (simple and dependency-free) or token-based (more accurate to the model's budget); the unit of counting is an orthogonal knob to the choice of splitter. In the examples below the count is character-based for clarity, but in production the same logic is run by tokens.

Cutting strategies: the catalog

Nine strategies - from the simplest to the smartest. First an overview table with ratings - you can read it as a cheat sheet and compare rows by eye. Under the table each strategy is unfolded: how it works, when to apply it, a step-by-step algorithm, and (for the main ones) working Python.

Cost is broken into three axes because they grow unevenly: Token cost - how many extra tokens the strategy produces; Time cost - the latency of cutting one document; Compute cost - the computational load. The ratings are relative (low / medium / high), not absolute numbers: they show the order of strategies relative to one another, not a benchmark on your hardware.

fixed-size: cut every N units

How it works. The simplest strategy: walk through the text and cut off exactly N units (characters or tokens) in a row, ignoring semantic boundaries. The size of each chunk is predictable; boundaries may land in the middle of a word or sentence.

When to apply. A quick prototype; uniform text without explicit structure; when you specifically care about size predictability. This is the baseline from which the other strategies push off (Pinecone, Chunking strategies).

Algorithm. Set the window size size; place the cursor i = 0; cut the substring text[i : i + size]; move the cursor i = i + size; repeat while i < len(text); the last chunk may be shorter than size - that is fine.

def fixed_size_chunks(text: str, size: int) -> list[str]:
    if size <= 0:
        raise ValueError("size must be positive")
    return [text[i:i + size] for i in range(0, len(text), size)]


if __name__ == "__main__":
    doc = "RAG mixes your documents into the prompt sent to the model. " \
          "The corpus is cut into chunks and each one is turned into a vector. " \
          "The vectors are stored in an index and the nearest by meaning are found."
    for n, chunk in enumerate(fixed_size_chunks(doc, 60)):
        print(n, repr(chunk))

Animation: the cuts land at equal intervals regardless of word boundaries - a cut can fall in the middle of a word.

sliding-window (overlap): a window with overlap

How it works. The same fixed-size window, but adjacent chunks overlap by overlap units: each next chunk begins not where the previous one ended but overlap earlier. This way a fact torn by a boundary lands in full in at least one chunk.

When to apply. When facts often fall on boundaries and losing boundary meaning is unacceptable. The price is text duplication: some tokens go into the index twice (Pinecone, Chunking strategies).

Algorithm. Set size and overlap (0 <= overlap < size); compute the step step = size - overlap; cut text[i : i + size] and move the cursor by step; the overlap tail of the previous chunk repeats at the start of the next.

def sliding_window_chunks(text: str, size: int, overlap: int) -> list[str]:
    if size <= 0:
        raise ValueError("size must be positive")
    if not 0 <= overlap < size:
        raise ValueError("overlap must satisfy 0 <= overlap < size")
    step = size - overlap
    chunks = []
    i = 0
    while i < len(text):
        chunks.append(text[i:i + size])
        i += step
    return chunks


if __name__ == "__main__":
    doc = "RAG mixes your documents into the prompt sent to the model. " \
          "The corpus is cut into chunks and each one is turned into a vector."
    for n, chunk in enumerate(sliding_window_chunks(doc, 50, 15)):
        print(n, repr(chunk))

Animation: the window moves along the text with step step = size - overlap; the overlap zone repeats in the adjacent chunk.

recursive (separator hierarchy): cut by separator priority

How it works. Cut by a prioritized list of separators - first by the largest (a double newline = a paragraph), then by smaller ones (a newline, a sentence, a space), until each piece fits the limit size. If a piece is still larger than the limit, the next separator is applied to it recursively. This is the default strategy in LangChain RecursiveCharacterTextSplitter (LangChain, RecursiveCharacterTextSplitter).

When to apply. The universal default for prose: neat boundaries by paragraphs and sentences, but no chunk exceeds the limit. Almost always better than bare fixed-size at the same low price.

def recursive_chunks(text: str, size: int,
                     separators: list[str] | None = None) -> list[str]:
    if separators is None:
        separators = ["\n\n", "\n", " ", ""]
    if len(text) <= size:
        return [text] if text else []

    sep = separators[0]
    rest = separators[1:]
    parts = list(text) if sep == "" else text.split(sep)

    chunks: list[str] = []
    buf = ""
    glue = "" if sep == "" else sep
    for part in parts:
        candidate = part if not buf else buf + glue + part
        if len(candidate) <= size:
            buf = candidate
            continue
        if buf:
            chunks.append(buf)
            buf = ""
        if len(part) <= size:
            buf = part
        elif rest:
            chunks.extend(recursive_chunks(part, size, rest))
        else:
            chunks.extend(part[i:i + size] for i in range(0, len(part), size))
    if buf:
        chunks.append(buf)
    return chunks


if __name__ == "__main__":
    doc = ("RAG mixes your documents into the prompt.\n\n"
           "The corpus is cut into chunks. Each chunk is embedded.\n\n"
           "The vectors are stored in an index and the nearest by meaning are found.")
    for n, chunk in enumerate(recursive_chunks(doc, 70)):
        print(n, repr(chunk))

This is a simplified but working implementation of the same idea as in LangChain RecursiveCharacterTextSplitter. A production splitter adds token counting and overlap; the logic of descending through the separators is the same.

Animation: first a cut by paragraphs; the pieces that did not fit by size are cut by sentences, then by words - a descent down the tree.

structure-aware: cut by sentence boundaries and structure

How it works. Cut not by a character count but by natural boundaries: sentence ends, Markdown headings, code AST elements. This best preserves meaning - a chunk coincides with a finished thought - but depends on the input format: you need reliable boundaries.

When to apply. There is explicit structure worth preserving: documentation with headings, code, transcripts with turns. Time cost is higher than fixed-size due to the segmentation or parsing stage (Pinecone, Chunking strategies).

import re


def markdown_header_chunks(text: str, size: int) -> list[dict]:
    sections: list[dict] = []
    heading = ""
    buf: list[str] = []

    def flush() -> None:
        body = "\n".join(buf).strip()
        if body:
            sections.append({"heading": heading, "text": body})

    for line in text.splitlines():
        m = re.match(r"^(#{1,6})\s+(.*)$", line)
        if m:
            flush()
            buf = []
            heading = m.group(2).strip()
        else:
            buf.append(line)
    flush()

    chunks: list[dict] = []
    for sec in sections:
        body = sec["text"]
        if len(body) <= size:
            chunks.append(sec)
            continue
        for i in range(0, len(body), size):
            chunks.append({"heading": sec["heading"], "text": body[i:i + size]})
    return chunks


if __name__ == "__main__":
    doc = ("# Vacation\n"
           "After 3 years of service you are entitled to 28 days of vacation.\n"
           "## Sick leave\n"
           "Sick leave is paid from the first day.")
    for n, chunk in enumerate(markdown_header_chunks(doc, 200)):
        print(n, chunk["heading"], repr(chunk["text"]))

This is a simplified version of the idea of LangChain MarkdownHeaderTextSplitter: cut by headings and put the heading level/text into the chunk metadata (LangChain, MarkdownHeaderTextSplitter). For sentences and code AST the wrapper is the same, only the parser changes.

Animation: the cuts land only at sentence ends and after a heading, never inside a phrase.

markdown / document-structure: cut by document headings

How it works. A particular but very common case of structure-aware: the document is already marked up with Markdown headings (#, ##, ...). Cut by headings, make each section a chunk, and put the heading level and text into the metadata. This way a chunk always coincides with a logical section, and the metadata shows its place in the document hierarchy.

When to apply. Documentation, READMEs, knowledge bases in Markdown - anything where the author already laid out the structure (LangChain, MarkdownHeaderTextSplitter). The working Python for this strategy is given above, in the structure-aware section (the markdown_header_chunks function).

Schematic: each section between # headings becomes its own chunk; the heading level rides into the metadata.

parent-document (small-to-big): index small, return large

How it works. Separate the unit of search from the unit of context. The document is cut twice: into small child chunks (precise hit on the query) and into large parent ones (broad context). Only the small chunks go into the vector index, but each stores a reference to its parent. At search time you find a small chunk, and into the model's context you slip its large parent.

When to apply. When small chunks win at search but the answer needs a wider piece than was found. This removes the main downside of small chunking - narrow context - without losing search precision (LangChain, ParentDocumentRetriever).

Schematic: small child chunks in the index reference a large parent; on a hit the parent is returned (small-to-big).

late chunking: embed the whole document, then pool by chunks

How it works. The ordinary pipeline first cuts, then embeds each chunk in isolation - and the chunk knows nothing about its neighbours. Late chunking reverses the order: first run the whole document through a long-context embedding model and get token representations that have already seen the whole text; and only then pool (average) these token vectors by the chunk boundaries. Each resulting chunk vector carries the context of the whole document (Gunther et al., 2024, "Late Chunking", arXiv:2409.04701).

When to apply. There is a long-context embedding model and it matters that a small chunk does not lose the resolution of coreferences ("the company", "it", "this product") beyond its fragment. The price is one pass of a heavy model over the whole document.

Schematic: the whole document is embedded in one pass; the token vectors are then averaged by the chunk boundaries (embed-whole-then-pool).

contextual retrieval: prepend context before embedding

How it works. Before embedding a chunk, prepend a short explanatory sentence that places the fragment in the context of the whole document. This explanation is generated by an LLM from the pair "document + chunk". What is embedded and indexed is the chunk-plus-context, so an isolated fragment stops being incomprehensible outside the document (Anthropic, Contextual Retrieval).

When to apply. When chunks often turn out uninformative in isolation from the document (tables, short bullet points, pronouns without an antecedent), and search quality matters more than the one-time price of an LLM enriching each chunk at indexing time.

Schematic: an LLM-generated explanatory context is prepended to each chunk before embedding (blurb prepended before embed).

semantic / LLM-based: cut by semantic shifts

How it works. Instead of counting characters or separators, the decision to cut is made by meaning. The typical approach: split the text into sentences, compute the embedding of each, and place a boundary where adjacent sentences diverge sharply in meaning (the cosine closeness between neighbours drops). A variant is to ask an LLM where to cut. The boundaries come out the most "semantic", but each candidate boundary costs a model call - hence high on all three cost axes (Pinecone, Chunking strategies).

When to apply. When boundary quality is critical and justifies the price. For most corpora recursive gives 90% of the benefit for a fraction of the price; semantic is taken when measurements show that boundaries are the bottleneck.

The threshold is chosen by measuring on a golden set, not guessed. Cosine as a measure of meaning closeness is taken apart in the Embedding chapter; semantic chunking relies on the same principle but applies it to cut boundaries rather than to search.

Animation: a boundary is placed where adjacent sentences diverge sharply in meaning (the cosine drops below a threshold), not at equal intervals.

How the catalog unfolds: progressive disclosure

The catalog is arranged as a semantic-zoom drill with two levels (there are no more than two):

• Level 0 - overview. Only the overview table of strategies with ratings is visible. This is the comparison layer: you run your eyes over the rows and pick a candidate. The static no-JS variant is exactly this table, fully readable without scripts.
• Level 1 - one strategy. A click or drill on a strategy row performs a semantic zoom into its panel: an unfolded "how it works" plus "when to apply" plus the step-by-step algorithm plus (for the basic ones) working Python plus a teaching animation of the cut for exactly this strategy. Exiting the zoom returns to the table.

The zoom does not go deeper than level 1. All animations run only when the panel is on screen (IntersectionObserver) and only under prefers-reduced-motion: no-preference; otherwise the end state is shown over the same DOM.

Size and overlap: the trade-off

Chunk size and overlap are two knobs of one trade-off. A small chunk: a precise hit on the query, but narrow context and the risk of tearing a fact. A large chunk: rich context, but blurred relevance and "lost in the middle" when assembling the prompt (Liu et al., 2023, arXiv:2307.03172). Overlap softens the main downside of cutting by boundaries - the loss of a boundary fact - at the price of token duplication (Pinecone, Chunking strategies).

The rule with no magic numbers: do not guess. Take a golden set of questions (see the Evaluation chapter), run two or three configurations (size/overlap/strategy), and compare the recall of the needed chunks. The strategy from the catalog above is the space of choices; measurement is the way to pick a point in it.

Link to the live example

In the live example from the overview, the source document doc.text is cut into three chunks by sentence boundaries - this is effectively a structure-aware cut by the end-of-sentence mark. The fromChar/toChar fields index exactly doc.text: each chunk is a precise slice of the source text, not a retelling of it. This is the Metadata-position from the drill-down in action: by it you can always reconstruct where the chunk came from and return a citation in the answer.

In this example the chunks do not overlap (overlap = 0) for clarity - this is pure structure-aware without sliding-window. Mentally add overlap: let the second chunk begin a few words earlier, capturing the tail of the first - that way a fact on the sentence boundary is not lost. This is exactly the transition from structure-aware to sliding-window from the catalog above.

Try it yourself

• Open the catalog at level 0 and compare the fixed-size and recursive rows: at the same low price, recursive gives neat boundaries by separators - that is why it is the default for prose.
• Drill into the sliding-window strategy and run its animation: watch how the window moves with step step = size - overlap and the overlap zone repeats in the adjacent chunk.
• Drill into recursive and walk the animation of descending through the separators ["\n\n", "\n", " ", ""]: the pieces that did not fit by paragraphs are cut by sentences, then by words.

What is next

The next stop of the route is Embedding: it takes these chunks and turns each into a numeric vector, where closeness = closeness of meaning (the same cosine that semantic chunking relies on).

Back to the route map