Chunking strategy for chatbot: stop splitting every 500 characters

Every RAG tutorial hands you the same chunker: slide a 500-token window over the text with a 50-token overlap, embed each window, done. It is a reasonable default for a pile of PDFs of unknown provenance. It is a bad default for markdown, because markdown comes with the seams already drawn.

When I added retrieval to this site’s chatbot, the retrieval plumbing took an afternoon. The chunking took considerably longer, and it is the part that decides whether the whole thing works. A vector index is only ever as good as the passages you put in it: the embedding model, the distance metric and the top-k are all downstream of the decision about where one idea stops and the next begins.

Why the fixed window might be wrong here#

A sliding window has no idea what it is cutting. Feed it a post and it will happily produce a chunk that starts three sentences into an explanation of one thing and ends two sentences into an explanation of another. Embed that and you get a vector that means neither. It will retrieve for both questions and answer neither well.

The overlap is a patch for this, not a fix. Overlap increases the chance that some window happens to contain a complete thought, at the cost of storing everything roughly twice.

Markdown does not need the patch. An author writing ## Why the fixed window might be wrong here has just told you, explicitly, that a self-contained idea starts on the next line and runs until the next heading. That is free structure, generated by a human who understood the material. Ignoring it to slide a fixed window over the result is throwing away the best signal in the document.

Strategy#

1. Clean before you measure. Strip what costs embedding tokens without carrying meaning: HTML comments, image tags, and the URL half of every link (the anchor text stays, since “the Opik feedback scores post” is meaningful and https://... is not). Long code blocks get truncated to their first few lines. This one is a judgement call: a chatbot answering “how did he do X” needs the prose explanation, and for the code it should send the visitor to the page. If you are indexing API documentation, invert that.

2. Split on h2 and h3. These are the author’s own outline. Track the heading trail so a chunk knows it belongs to Experience > Falabella India > Senior Architect, and skip the h1, which is the document title and already lives in metadata.

3. Fold the runts. A section under 320 characters does not become a chunk. It gets appended to the previous one. A two-line paragraph embedded on its own is a vector with almost no signal: it retrieves for the wrong things, and when it does retrieve correctly it hands the model a fragment too small to answer from. Its heading folds in with it, so no wording is ever lost.

4. Pack the oversized ones down a ladder. A section over 1400 characters splits at paragraph boundaries. A single paragraph over the cap splits at sentence boundaries. A single “sentence” over the cap, which in practice means a bullet list nobody punctuated, splits at line boundaries, and then finally at raw character count. Each rung is uglier than the last and each is hit less often. Having the ugly rung matters: it is what stops one badly formatted table from producing a chunk that silently exceeds the model’s input limit.

Every chunk that comes out of a split repeats its heading. A chunk that does not say what it is about is a chunk that retrieves for the wrong question.

The result on my site: 104 chunks from about 73,000 characters of posts, weekly notes, resume and project descriptions. Median 676 characters, longest 1,390, shortest 85.

Where the thresholds come from#

1400 characters. This is the only number derived rather than chosen. bge-base-en-v1.5 truncates input at 512 tokens, and English prose runs about 4 characters per token, so the ceiling is roughly 2,000 characters. I prepend the document title to every chunk before embedding, and I would rather have headroom than discover the truncation empirically, because that is another silent failure: the model does not reject an over-long input, it just quietly stops reading. 1400 leaves comfortable room.

320 characters for the fold. Chosen, not derived. It is about a short paragraph. Below that, a chunk is a fragment.

80 characters to drop entirely. A section that survives cleaning at under 80 characters is a stub heading or a one-line aside. It is noise in the index.

I would not defend the exact values of the last two. I would defend having them: the failure mode of no minimum is an index full of tiny high-variance vectors that pollute every query.

Two things that bit me#

Fenced code confuses a naive splitter. A shell example containing ## install dependencies is a comment, not a heading. If your splitter does not track whether it is inside a fence, that comment starts a new chunk in the middle of a code block. Tracking fence state is four lines and I have now written it twice, having not learned the first time.

Line endings are load bearing. My splitter worked on every post and produced exactly one enormous chunk for my resume, which is the one file in the repo with Windows line endings. In JavaScript, $ outside multiline mode does not match before a trailing carriage return, and . will not consume one, so ^###\s+(.+)$ simply fails on ### Heading\r. No error. The heading stops being a heading and the entire document becomes one chunk.

Normalise line endings as the first thing your cleaner does. I now think of this as a general rule for text pipelines: normalise the invisible characters before any logic depends on them, because every bug in that class is silent, and a silent bug in a chunker does not surface as a crash. It surfaces as answers that are slightly worse than they should be, months later, with no way to tell.

What I would try next#

Chunking well is necessary and it is not sufficient. Three things I deliberately left out, in the order I would add them:

• Contextual retrieval. Prepend a one-line, LLM-generated summary of the parent document to each chunk before embedding, so a chunk about “the fallback path” knows which system’s fallback path it describes. This is the highest-value addition I am not doing yet, and it costs one cheap LLM call per chunk at index time.
• Hybrid search. Vector similarity is bad at exact tokens. Nobody asks my site about CAPTURE_SYNC_TOKEN, but on a documentation site that class of query is most of the traffic, and keyword search plus vector search beats either alone.
• Reranking. Retrieve twenty, rerank to four with a cross-encoder. Worth it when top-k precision is the bottleneck. On a corpus of 104 chunks it would be solving a problem I do not have.

Reach for a fixed-size window when your input has no structure. When it does have structure, and markdown always does, the chunker’s job is to notice it rather than to overwrite it.