Chris Collins

Tools · search

Analyzer and inverted-index playground

Build an Elasticsearch analyzer from char filters, a tokenizer and token filters and watch the token stream change stage by stage — terms, positions, offsets, types and position lengths, drawn as a token graph — then index documents to see the postings, field-length norms and phrase/slop matching, and copy the _analyze request and index settings.

Added

How to use it

How to use it

Pick a preset (standard, english, simple, whitespace, keyword, stop, autocomplete, synonyms, product codes, paths, HTML, shingles) or edit the chain: add, reorder and configure char filters, the tokenizer and token filters. Everything re-runs as you type.

Analyze. Each stage shows its tokens as chips (or a table: term, position, start and end offset, type, positionLength), with new or changed tokens tinted and the change in count. Hover or tap a token to highlight the characters it came from in the original text — offsets survive char filters, so a token from <b>happy</b> points into the HTML. The last card draws the token graph: positions are nodes and each token is an arc; synonyms that span several positions (ny ↔ new york) show why synonym_graph and positionLength exist.

Index & postings. Documents (one per line) are indexed with the analyzer. The inverted index lists every term with its document frequency, total term frequency, BM25 idf and postings (d3: 2 @ [1, 5] = document 3, frequency 2, at positions 1 and 5; hover for offsets). The norms table shows each field’s length, the length Lucene counts (stacked tokens at the same position are not counted: discount_overlaps), the one-byte norm and the length it decodes to.

Phrase & slop. Type a phrase and drag the slop. Each document’s positions are drawn as cells; query terms are outlined and matched windows filled, with the sloppy frequency Σ 1/(1 + distance) and the BM25 score and explanation. Holes left by stop words stay holes, so "fox on a box" matches "fox in the box".

Elasticsearch JSON. The index settings (analysis with named definitions for anything configured) and the equivalent _analyze request, with the response Elasticsearch would give. Paste settings or an _analyze body to load an analyzer; with several analyzers, pick one.

Components

Char filters

html_strip (tags, comments, script/style removed; block tags become newlines; entities decoded; escaped_tags), mapping (longest match wins), pattern_replace (Java-style $1 groups)

Tokenizers

standard (Unicode UAX #29 word boundaries, as Lucene’s StandardTokenizer: CJK ideographs one per token, katakana runs, emoji, Hangul, Southeast Asian runs; max_token_length splits long tokens), classic (emails, hosts, acronyms, company names, product numbers), whitespace, letter, lowercase, keyword, pattern (split or capture group), char_group, ngram, edge_ngram (with token_chars), path_hierarchy (all prefixes at one position; reverse, skip)

Token filters

lowercase (per code point, as Lucene: Greek final sigma stays σ, İ → i), uppercase, asciifolding (preserve_original), stop (english = Lucene’s 33 words, or your list; leaves position holes), porter_stem, stemmer (english = Porter, porter2 = Snowball English, minimal_english, possessive_english), snowball, keyword_marker, synonym and synonym_graph (Solr and WordNet formats, expand), word_delimiter_graph, shingle (filler_token for holes), edge_ngram, ngram, unique, length, trim, truncate, reverse, elision, classic, decimal_digit

The tests check the tokenizers against the examples in the Elasticsearch reference (the standard and classic tokenizers on "The 2 QUICK Brown-Foxes jumped over the lazy dog’s bone.", max_token_length: 5, the ngram and edge_ngram examples, path_hierarchy, the html_strip, mapping and pattern_replace examples, word_delimiter_graph on "Neil’s-Super-Duper-XL500—​42+AutoCoder", shingles), Porter’s stemmer against the examples in M. F. Porter, An algorithm for suffix stripping (1980), and Porter2 against the Snowball English sample vocabulary.

Limits

  • This is a model of the Lucene analyzers in JavaScript, not Lucene. UAX #29 is implemented for the cases above (letters, digits, mid-word punctuation, underscores, scripts, emoji); obscure rules (Hebrew quotes, some Extend/Format cases) may differ.

  • ASCII folding uses Unicode decomposition plus a table for letters without one (æ, ø, ł, ß, þ…) and typographic punctuation; Lucene’s own table has about 1,400 entries and covers more symbols.

  • kstem (light_english) needs its 30,000-word dictionary and is not included; neither are the non-English stemmers, ICU, the Kuromoji/Nori/SmartCN analyzers, or stop lists other than english (give your own words instead).

  • word_delimiter (the legacy, non-graph filter) is modelled like the graph one but stacks parts at one position. synonym_graph follows Lucene’s layout of side paths; very complex rule sets (overlapping multi-word rules) may be laid out differently.

  • Phrase matching follows Lucene’s sloppy matcher without its special handling of repeated terms ("to be or not to be"), so slop with repeated words can count matches differently.

Related: Relevance and scoring explainer (BM25 on these postings), Query DSL builder (run match, phrase and fuzzy queries over a sample index), Regex tester (for pattern tokenizers and char filters), Unicode inspector, TF-IDF and BM25 basics.

Linking to an example

Links can be written by hand. Add the tab after #: analyze, index, phrase or json.

Parameter Meaning Example

preset

An analyzer preset: standard, simple, whitespace, keyword, stop, english, autocomplete, synonyms, product_codes, path, html_folded, shingles

preset=english

text

The text to analyze

text=Running+dogs

q

The phrase on the Phrase tab

q=quick+fox

slop

Its slop, 0–10

slop=1

example

One of the presets below; parameters given with it override it

example=synonym-graph
example= Shows

english

The english analyzer: possessives, stop words and Porter stems (open)

synonym-graph

Multi-word synonyms with synonym_graph: "ny" spans two positions (open)

html

html_strip then ASCII folding, with offsets mapped back into the HTML (open)

autocomplete

Edge n-grams for search-as-you-type (open)

sku

word_delimiter_graph splitting product codes, with catenations (open)

unicode

UAX #29 on CJK, emoji, Greek final sigma, Turkish İ, Hangul and Thai (open)

path

path_hierarchy: every ancestor path at one position (open)

shingles

Word bigrams (shingles) with their position lengths (open)

phrase-slop

"quick fox" with slop 1 against four documents (open)

reversed-slop

"fox quick" needs slop 2: swapping two terms costs two moves (open)

Runs entirely in your browser; nothing you type leaves this page.