Chris Collins

Tools · search

Relevance and scoring explainer

See why documents rank where they do: BM25 term by term like Elasticsearch's _explain, with k1 and b sliders that re-rank live, classic TF-IDF side by side, multi-field strategies and boosts, function_score (field_value_factor, decay curves, a recency script), and hybrid lexical + vector ranking with reciprocal rank fusion or a normalised linear blend.

Added

Works with

How to use it

How to use it

Choose a document set — a product catalogue, articles about search, or short vs long fields (made to show length normalization and saturation) — or edit your own as JSON. Type a query; drag k1 and b. Everything re-ranks as you go.

BM25 explained. Pick a field and OR/AND. The query terms appear with their document frequency and idf; the table ranks the matches with their BM25 score, how far each moved from the defaults (k1 = 1.2, b = 0.75), and its rank and score under classic TF-IDF. Click a row for the per-term breakdown (tf, df/N, idf, dl, avgdl, tf/(tf + K), contribution) and the explanation tree in the shape _explain returns.

Multi-field. Tick fields and set boosts (title^3), choose best_fields, most_fields or cross_fields, a tie_breaker, and an optional phrase boost (a should match_phrase). The generated query, its Lucene form and the ranking under all three types are shown side by side.

Function score. Combine the text score with field_value_factor (any modifier: log1p, sqrt, ln2p…), a gauss/exp/linear decay on a number or date (origin, scale, offset, decay), and a recency script_score (half / (half + age in days)). The charts draw the decay shapes with every document placed on the curve, and the modifier curve; the table shows text score, function value, final score and rank change. "Now" is fixed at 2026-10-15 so results are reproducible.

Hybrid (RRF). The lexical query (BM25) and a vector query (toy 8-concept embeddings, so "earbuds for the gym" finds sports earbuds without the word) produce two ranked lists, fused by reciprocal rank fusion (Σ 1/(k + rank), Cormack, Clarke and Büttcher, SIGIR 2009) or by a linear blend of min–max normalised scores. Adjust k, the window and α; the equivalent Elasticsearch retriever JSON is shown.

Curves. Term-frequency saturation for several k1 against classic √tf; the length-normalization factor against dl/avgdl for several b; and BM25’s idf against classic idf.

The formulas

BM25 (Lucene ≥ 8)

score = boost · idf · tf / (tf + k1 · (1 − b + b · dl / avgdl)); idf = ln(1 + (N − n + 0.5) / (n + 0.5)). Lucene’s BM25Similarity dropped the constant (k1 + 1) numerator factor in 8.0 (LUCENE-8563): it does not change rankings.

BM25 (Elasticsearch)

Elasticsearch 7 and 8 use LegacyBM25Similarity, which multiplies the boost by (k1 + 1), so _explain shows "boost 2.2" and scores equal the textbook formula. The Elasticsearch setting here (the default) reproduces the _explain example in the Elasticsearch reference exactly: 1.6943598 = 2.2 × 1.3862944 × 0.5555556.

Field length

dl is stored in one byte per document (Lucene SmallFloat.intToByte4): exact up to 23 terms, then 4 significant bits — a 100-term field is scored as 96. avgdl = sumTotalTermFreq / docCount is exact.

Classic TF-IDF

Lucene ClassicSimilarity (since 7.0, without queryNorm or coord): score = boost · idf · √tf · 1/√dl, idf = 1 + lnN + 1) / (n + 1.

Decay

gauss: exp(−d² / 2σ²) with σ² = −scale² / (2 ln decay); exp: e^(λd), λ = ln(decay)/scale; linear: max(0, (s − d)/s), s = scale/(1 − decay); d = max(0, |value − origin| − offset) — the Elasticsearch function_score reference.

RRF

score(d) = Σ 1/(k + rank_i(d)), ranks from 1, k = rank_constant (default 60).

Limits

  • One shard, no deletions: document frequencies are exact. In a real cluster idf is per shard unless you use dfs_query_then_fetch, which can reorder close scores.

  • Scores are computed in float32 like Lucene; sums can differ from Elasticsearch in the last digit.

  • cross_fields blends document frequencies as the largest across the fields (an approximation of Elasticsearch’s BlendedTermQuery).

  • The vectors are a hand-made 8-dimension toy for teaching, not a language model; real hybrid search uses a trained embedding model and an approximate (HNSW) index.

  • Only the script forms shown are supported (a small Painless subset).

Related: Analyzer and inverted-index playground, Query DSL builder and explainer, TF-IDF and BM25 from first principles, Attention and KV cache, Statistics.

Linking to an example

Links can be written by hand. Add the tab after #: explain, fields, functions, hybrid or curves.

Parameter Meaning Example

set

Document set: products, articles, shortlong

set=shortlong

q

The query

q=lucene

field

The field on the BM25 tab

field=body

k1

BM25 k1 (0–3)

k1=2

b

BM25 b (0–1)

b=0.5

v

Score variant: es (× (k1+1), Elasticsearch) or lucene

v=lucene

op

Operator on the BM25 tab: or or and

op=and

example

One of the presets below; parameters given with it override it

example=saturation
example= Shows

saturation

Term-frequency saturation: "Lucene" five times barely beats once (open)

length-norm

b = 1: full length normalization, the one-word title wins (open)

no-length-norm

b = 0: length ignored (open)

high-k1

k1 = 3: repeated terms keep paying off for longer (open)

cross-fields

best_fields vs cross_fields for a query spread over title and body (open)

phrase-boost

A should match_phrase adds a bonus when the words are adjacent (open)

popularity

field_value_factor: log1p(reviews_count) multiplies the text score (open)

recency

A gauss decay on the publish date: newer articles rise (open)

hybrid-rrf

Lexical BM25 + toy vectors fused with RRF, k = 60 (open)

hybrid-linear

The same with a min–max linear combination, α = 0.3 (open)

Runs entirely in your browser; nothing you type leaves this page.