Chris Collins

Tools · ai

RAG chunking playground

Drop in text, Markdown, HTML or PDF and chunk it five ways — fixed token windows with overlap, sentences, LangChain's recursive splitter, Markdown headings and semantic breakpoints — see every boundary and overlap on the text, then compare BM25, embedding and hybrid retrieval across the chunkers with recall@k and MRR against passages you mark.

Added

How to use it

How you cut documents into chunks decides what a retrieval-augmented system can find: too small and an answer is split across pieces, too big and every hit drags a page of noise into the prompt. This page chunks the same documents several ways, draws the boundaries on the text, and measures retrieval for each chunker against answers you mark. The text stays in your browser; only the libraries and the embedding model are downloaded.

How to use it

Documents. Start with the sample corpus — a Markdown runbook, a plain-text essay on lighthouse optics and an HTML FAQ, with eleven questions whose answers are already marked — or open, drop or paste your own .txt, .md, .html and .pdf files. HTML is turned into text with its headings as Markdown # lines (so the Markdown chunker can use them) and list items as `- ` lines; PDF text is extracted page by page with pdf.js (loaded from cdnjs when you open a PDF). Your documents, queries and marks are remembered in this browser; Export JSON saves the whole project and Import JSON brings it back.

Chunks. Set the chunk size and overlap and what they are measured in: characters (code points, as Python’s len), words, tokens (≈ cl100k, see below) or exact cl100k_base tokens. Then pick the chunkers to compare:

Chunker What it does

Fixed windows

Windows of exactly size units, each starting size − overlap units after the last — LangChain’s TokenTextSplitter when the unit is tokens. Cuts mid-sentence, even mid-word.

Sentences

Whole sentences packed up to the size, carrying the last n sentences (default 1) into the next chunk. A sentence longer than the size becomes a chunk of its own.

Recursive (LangChain)

LangChain’s RecursiveCharacterTextSplitter: split on the first separator that occurs ("\n\n", "\n", " ", "" by default), merge pieces up to the size with up to overlap carried over, and split any piece that is still too long with the next separator.

Markdown headings

One chunk per section at headings up to the chosen level (code fences are skipped), with the heading path (Runbook › Alerts › Replication lag) kept as metadata, like LangChain’s MarkdownHeaderTextSplitter. Long sections can be split again recursively, and the path can be indexed in front of each chunk.

Semantic breakpoints

Greg Kamradt’s method, as LangChain’s SemanticChunker: embed each sentence together with one neighbour each side, measure the cosine distance between consecutive ones, and cut where it is above the 95th percentile (or mean + 3σ, or mean + 1.5 × IQR). Needs the embedding model.

The text shows each chunk in its own colour with its number at its start; striped text is in two chunks (the overlap) and grey text in none (whitespace the splitter dropped, or a heading-only Markdown section). Tap a chunk to read it on its own. The table compares every chunker: how many chunks, their mean, median and range of lengths, how much text is duplicated by overlap, and how much is covered. The histogram shows the selected chunker’s lengths against the size; for the semantic chunker a chart shows the distances between sentences, the threshold and where it cut. Full screen (or F) gives the text the whole screen.

Retrieval. Each query is run against the chunks of every chunker with

  • BM25 — Elasticsearch’s scoring (k1 = 1.2, b = 0.75, english or standard analyzer) from the site’s search engine model;

  • Embeddings — cosine similarity between all-MiniLM-L6-v2 vectors of the query and each chunk, computed in your browser (the model is downloaded once, about 23 MB);

  • Hybrid — the two rankings fused with reciprocal rank fusion, score = Σ 1 / (k + rank), k = 60 (Cormack, Clarke & Büttcher 2009).

To judge a query, select it, select the answer in the document text and press Mark selection relevant; a query can have several passages, in several documents. Because chunkers cut the same text differently, the scores are measured on those passages, not on chunk ids:

  • a chunk answers a passage when it covers at least half of the passage’s characters (the A chunk answers a passage if it covers control: any overlap, at least half, or all of it);

  • Recall@k is the share of a query’s passages answered by some chunk in the top k;

  • MRR is the mean of 1 / rank of the first chunk that answers any passage (over the top 100);

  • Context@k is the length of the top k chunks — what you would paste into the prompt;

  • Precision is the share of those characters that lie inside marked passages.

The table shows the means over the judged queries (the best recall in each column in green); tap a chunker to see its top k for the selected query, ticked when they answer, and underlined in the text with their ranks.

Tokens. The ≈ tokens unit runs cl100k_base’s own pre-tokeniser regex (from OpenAI’s tiktoken) and counts each piece of up to nine characters (a word with its leading space, a run of punctuation, up to three digits) as one token and longer pieces as one per nine characters. It is a rule of thumb: on the three sample documents and LangChain’s example text it lands within 5% of tiktoken’s count (−2% to +5%), but code, numbers-heavy text and other languages can be far off. Choose tokens (cl100k exact) to download the real vocabulary (1.1 MB from jsDelivr, as in the tokenizer explorer) and count exactly; the fixed windows then cut at real token boundaries.

Limits

  • The sentence splitter is a set of rules (stops followed by a capital, common abbreviations, list items and headings), not a trained model; it will mis-split some text. LangChain’s SemanticChunker splits sentences with a simpler regex, so its chunks can differ slightly.

  • all-MiniLM-L6-v2 reads at most 256 word pieces; longer chunks are embedded from their start only, which handicaps big chunks in the embedding columns. It is a small English model, not the model your system uses.

  • BM25 statistics are computed per chunker over that chunker’s chunks, so idf differs between chunkers — as it would in separate indexes.

  • Recall is about whether one chunk covers enough of a passage; an answer spread over two chunks that each hold less than the threshold does not count.

  • HTML conversion is a simple tag-to-text pass (no CSS, no layout); PDF text comes out in the order pdf.js reports it, which for multi-column pages may be wrong. Scanned PDFs have no text.

  • Large corpora (several MB) work but re-chunk on every settings change; embedding thousands of chunks on the CPU takes minutes.

Checked against

The tests (tests/tools/rag-chunking.test.js) check the recursive splitter against LangChain’s own test suite (test_iterative_text_splitter: chunk size 10, overlap 1, 18 expected chunks) and the first chunks of its documentation example (the 2022 State of the Union with chunk_size=100, chunk_overlap=20), the merge step against CharacterTextSplitter’s `"foo bar baz 123" example, the token windows against split_text_on_tokens, exact token counts against tiktoken’s cl100k_base ids, and percentiles against NumPy’s linear interpolation.

Sources: LangChain langchain-text-splitters (RecursiveCharacterTextSplitter, CharacterTextSplitter, TokenTextSplitter, MarkdownHeaderTextSplitter) and langchain_experimental.text_splitter.SemanticChunker; G. Kamradt, “The 5 levels of text splitting for retrieval” (2024); S. Robertson & H. Zaragoza, “The probabilistic relevance framework: BM25 and beyond” (2009); G. Cormack, C. Clarke & S. Büttcher, “Reciprocal rank fusion outperforms Condorcet and individual rank learning methods” (SIGIR 2009); N. Reimers & I. Gurevych, “Sentence-BERT” (2019) for all-MiniLM-L6-v2.

Related: Tokenizer explorer, Embedding explorer, Vector search, Relevance and scoring, Ranking metrics and Eval harness.

Linking to an example

A link opens the tool with the given settings: /tools/rag-chunking/?<parameters>#<tab>, where the tab is docs, chunks or retrieve. Share link builds one from what is on screen; your own documents are never put in a link (a link to them opens with the recipient’s documents).

Parameter Meaning Example

example

a preset (below); other parameters in the same link override it

tiny-chunks

corpus

sample, sotu (LangChain’s example text) or custom

sample

size, overlap

chunk size and overlap

400, 50

unit

chars, words, tokens (≈) or cl100k

tokens

on

chunkers to compare, comma separated: fixed, sentence, recursive, markdown, semantic

recursive,markdown

ov

sentences carried into the next sentence chunk

1

level, mdsplit, ctx

Markdown: deepest heading level, split long sections again, index the heading path

2, 1, 1

seps

recursive separators as a JSON list

["\n\n","\n"," ",""]

sem, amt

semantic breakpoint rule (percentile, standard_deviation, interquartile) and amount

percentile, 90

k, theta

top k, and the share of a passage a chunk must cover (0, 0.5, 1)

5, 1

an, rrfk

BM25 analyzer (english, standard) and RRF k

standard, 60

emb, method

embeddings on (1), and the method shown per query (bm25, emb, hybrid)

1, hybrid

chunker, doc, q

the chunker, document (0-based) and query shown

markdown, 0, 2

Presets for example=:

Name Tab Shows

sample

retrieve

The sample corpus with four chunkers at 400 characters (open)

tiny-chunks

retrieve

120-character chunks and a passage must be covered whole: recall falls apart (open)

big-chunks

retrieve

1,500-character chunks: recall is easy, but every answer costs far more context (open)

no-overlap

chunks

60-token fixed windows with no overlap cut sentences mid-word (open)

tokens

chunks

128-token windows with 16 tokens of overlap (open)

markdown-context

retrieve

Markdown sections with their heading path indexed against the recursive splitter (open)

langchain-docs

chunks

LangChain’s documentation example, chunk_size=100, chunk_overlap=20, on the State of the Union (open)

semantic

chunks

Semantic breakpoints at the 80th percentile of the distances on the lighthouse essay (downloads the model) (open)

For example, ?example=sample&size=250&overlap=0&k=1#retrieve asks how often the single best 250-character chunk holds the answer.

Runs entirely in your browser — network requests go only to the services named on this page.