Tokenizer explorer

Compare. Pick a sample (or type) and tick the tokenizers to compare. Each card shows the token count, tokens per word, characters and bytes per token, and the cost at the price you enter (dollars per million tokens; the card also gives the price per 1k and the cost of 1,000 such texts). Tokens are drawn as coloured boxes with spaces shown as ·, tabs as → and newlines as ↵; tap one for its id and bytes. Ids switches to the id list (with a copy button). A dashed red token is not a whole character (its bytes are part of a UTF-8 sequence that continues in the next token, shown as ‹F0 9F›); a hatched token is unknown to the vocabulary ([UNK], <unk>). Recognise special tokens treats <|endoftext|>, [MASK] and friends as control tokens instead of text.

Vocabularies download the first time a tokenizer is ticked — tiktoken’s ranks from jsDelivr (the js-tiktoken 1.0.21 package), the others from the Hugging Face Hub — and the browser caches them. Your text never leaves the page.

Bytes. Every character with its code point and UTF-8 bytes, each byte coloured by the token that holds it, so you can see an emoji or a Devanagari syllable split across tokens (dashed red boxes) and Llama 2’s byte fallback (one <0xNN> token per byte for characters it cannot spell). For the tiktoken encodings the panel below lists the pre-tokeniser’s pieces — the regular expression that splits words (with their leading space), digit groups of up to three, punctuation runs and whitespace before BPE runs — and tapping a piece shows its bytes being merged, lowest rank first, until no ranked pair is left.

Normalisation. Unicode NFC, NFD, NFKC and NFKD, lowercasing and accent stripping applied to your text, with every code point before and after (combining marks tinted, changed ones outlined), the code-point and byte counts of each form, what BERT’s and T5’s normalisers produce, and — for the tokenizers already loaded on Compare — the token count before and after.

Train a BPE. Learn byte-pair-encoding merges from a few lines of text as in Algorithm 1 of Sennrich, Haddow & Birch (2016): count adjacent symbol pairs over all words (weighted by word frequency), merge the most frequent, repeat. Next, Back, Play and Restart walk through the merges; each step shows the pair counts that chose it, every word’s current segmentation, the merge table (its order is the rank), and a chart of the corpus length in tokens and the symbols in use against the number of merges. Words can end with </w> (the paper) or start with ▁ (SentencePiece style). The learned merges segment any new text you type, and the Tiny BPE column on Compare uses the model at the current step.

Pitfalls. Leading spaces and capitals, numbers, code and whitespace, non-Latin scripts and CJK, emoji and joiner sequences, accents and normalisation, special tokens in user text, rare words, and why token counts differ between models — each with a Show me button.

The tests (tests/tools/tokenizer.test.js) compare this page’s tokenizers with ids produced by tiktoken 0.14 and Hugging Face tokenizers 0.23 for ten strings — English, leading spaces and tabs, numbers and dates, Python, accented Latin, Japanese and Chinese, emoji with a zero-width joiner, Arabic, Hindi and Russian, rare words, and compatibility characters — and reproduce the ten merges of Sennrich et al.'s example (e s, es t, est </w>, l o, lo w, n e, ne w, new est</w>, low </w>, w i).

Related: Attention and the KV cache (what each token costs in memory), LLM serving (tokens per second and cost per token), Unicode inspector, AI economics and The maths of machine learning.

A link opens the tool with the given settings: /tools/tokenizer/?<parameters>#<tab>, where the tab is compare, bytes, normalise, train or pitfalls. Share link (top right) builds one from whatever is on screen.

Presets for example=:

For example, https://geekychris.github.io/tools/tokenizer/?text=Supercalifragilistic%20expialidocious&tk=cl100k,o200k,bert#compare compares one long word in three tokenizers.