Ranking metrics workbench

Offline relevance evaluation in the browser: judgments and runs in, the measures your search team argues about out, computed the way trec_eval computes them (the numbers match it to the last printed digit), with the step-by-step arithmetic for any one ranking, significance tests for “is B really better than A?”, and an interleaving simulator for the online version of that question. Nothing is uploaded.

How to use it

Data. Paste or open the judgments (qrels) and one or two runs.

Judgments: TREC qrels, qid 0 docno grade one per line (the second column is ignored), or CSV/TSV qid,docno,grade. A header row may name the columns (query, doc, rel/grade/label). Grades are integers (0 = not relevant, higher = more relevant; negative counts as not relevant).
Runs: TREC qid Q0 docno rank score tag, or CSV/TSV qid,docno,score; with a header you may give rank instead of score (lower ranks first). A bare qid docno list keeps its order.
Load an example fills everything in; Generate pooled collection makes a synthetic TREC-style collection: queries 401…, two systems (bm25 and a less noisy bm25+rerank) returning 20 documents each, judged to pool depth 10 plus a few relevant documents neither found, so deep results are often unjudged — as in real pooled collections.
Evaluation settings: the cut-offs k (up to six), the grade that counts as relevant for the binary measures (trec_eval -l, default 1), the maximum grade for ERR (gdeval uses 4), the depth (-M: only the top N results count), average over every judged query (-c: a query the run does not answer scores 0 instead of being left out) and judged documents only (-J: unjudged results are removed before scoring — “condensed lists”, Sakai 2007).

The rules are trec_eval’s. Results are ordered by score descending, ties by docno descending (the rank column is ignored); a document retrieved twice for a query is an error (the first is kept); unjudged documents count as non-relevant; queries in the run without judgments are skipped and listed.

Metrics. The mean of every measure for A and B with the difference, a per-query table (tap a query to see its ranking as coloured grades — ? is unjudged — and to open it in One ranking), and the same numbers as trec_eval’s text output (averages or per query), ready to diff.

Measure	Definition
P@k	relevant in the top k ÷ k (always k, even if fewer were retrieved)
R@k	relevant in the top k ÷ all relevant for the query
AP / MAP	the mean of P@i at the rank i of each relevant document; relevant documents not retrieved add 0. MAP is the mean over queries
RR / MRR	1 ÷ rank of the first relevant document (0 if none)
R-precision	P@R where R is the number of relevant documents (the point where precision = recall)
nDCG@k	Σ gain(gᵢ)/log₂(i + 1) to depth k, divided by the same sum for the ideal ordering of all judged grades (Järvelin & Kekäläinen 2002). lin: gain = grade (trec_eval). exp: gain = 2^g − 1 (Burges et al. 2005)
ERR@k	expected reciprocal rank under a cascade user, Σ (1/r)·R_r·Π_{i<r}(1 − R_i), R = (2^g − 1)/2^gmax (Chapelle, Metzler, Zhang & Grinspan 2009)

Measure

Definition

P@k

relevant in the top k ÷ k (always k, even if fewer were retrieved)

R@k

relevant in the top k ÷ all relevant for the query

AP / MAP

the mean of P@i at the rank i of each relevant document; relevant documents not retrieved add 0. MAP is the mean over queries

RR / MRR

1 ÷ rank of the first relevant document (0 if none)

R-precision

P@R where R is the number of relevant documents (the point where precision = recall)

nDCG@k

Σ gain(gᵢ)/log₂(i + 1) to depth k, divided by the same sum for the ideal ordering of all judged grades (Järvelin & Kekäläinen 2002). lin: gain = grade (trec_eval). exp: gain = 2^g − 1 (Burges et al. 2005)

ERR@k

expected reciprocal rank under a cascade user, Σ (1/r)·R_r·Π_{i<r}(1 − R_i), R = (2^g − 1)/2^gmax (Chapelle, Metzler, Zhang & Grinspan 2009)

Linear or exponential gain? trec_eval (ndcg, ndcg_cut), pytrec_eval, ir_measures' default and scikit-learn’s ndcg_score use the grade itself as the gain. gdeval.pl (the TREC Web track’s script), RankLib, LightGBM and XGBoost ranking objectives, TensorFlow Ranking and the Elasticsearch/OpenSearch rank_eval DCG use 2^grade − 1. With binary judgments they agree; with grades 0–3 a “3” is worth 3 “1”s in one and 7 in the other, so two runs can swap places. The table on the One ranking tab lists them; always say which you report.

One ranking. Type the grades of a ranked list (and of relevant documents it missed), choose k, and every measure is worked rank by rank: P@i, each AP term, the discount, linear and exponential gains with running DCG against the ideal, and ERR’s satisfaction probability, the chance of reaching each rank and its term. The chart shows how each measure accumulates down the list — ERR saturates after one excellent hit, AP and nDCG keep counting, RR sees only the first hit.

Compare. Pick a measure; the two runs are paired by query (only queries both answer) and the differences d = B − A tested four ways, two-sided:

Paired t-test — t, df, p and the confidence interval of the mean difference.
Wilcoxon signed-rank — zero differences dropped; exact null distribution when n ≤ 50 and no tied |d|, otherwise the normal approximation with tie correction (SciPy’s defaults).
Randomisation test — Fisher’s sign-flip test of the mean (Smucker, Allan & Carterette 2007): exact over all 2ⁿ sign patterns for n ≤ 20 queries, else B random patterns with the seed.
Bootstrap — B resamples of the queries; percentile CI and the shifted-bootstrap p-value (Efron & Tibshirani 1993; Sakai 2006).

Plots show the sorted per-query differences, A against B per query, the randomisation distribution with the tail that makes the p-value, and the bootstrap distribution with its CI. The same seed always gives the same numbers.

Interleaving. Team-draft interleaving (Radlinski, Kurup & Joachims 2008) of runs A and B with simulated users: each impression picks a query, merges the two lists (a coin decides who drafts first each round), and a cascade click model clicks from the judgments — perfect, navigational and informational from Hofmann, Whiteson & de Rijke (2011) with grades 0, 1 and 2+, and random as a control. The ranker whose documents get more clicks wins the impression. You get wins, ties, the share of decided impressions B wins with a sign-test p-value, how that share settles as impressions accumulate, and individual impressions with team, grade and clicks.

Limits

Implements the trec_eval measures listed here (not all ~100 of -m all_trec, e.g. no bpref, infAP, or interpolated precision); per-level custom nDCG gains are not offered.
ERR and the exponential nDCG are computed on the same ordering and depth as everything else; gdeval.pl reports them only at depth 20 for numeric topics with at least one relevant document.
Up to about 100,000 lines per file stay responsive. Randomisation and bootstrap with large B on hundreds of queries take a second.
The click models are simple cascades; real users also show position bias and attractiveness that differs from relevance. The synthetic collection is random data, not a real benchmark.

Checked against

The tests run the tool’s datasets through genuine reference implementations: trec_eval 10.0 (every measure above, per query and averaged, for 25 pooled queries, the tie example and the textbook examples), gdeval.pl 1.2 from the TREC 2010 Web track (nDCG@20 with exponential gain and ERR@20), and SciPy 1.17 (ttest_rel, wilcoxon exact and approximate, permutation_test, t quantiles). Worked examples: Croft, Metzler & Strohman, Search Engines: Information Retrieval in Practice (2010) §8.4 — AP 0.71 and 0.33 for the two rankings, MAP 0.53 for the two queries — and Wikipedia’s DCG example (DCG₆ 6.861, IDCG 8.740, nDCG 0.785).

Sources: Manning, Raghavan & Schütze, Introduction to Information Retrieval ch. 8; Voorhees & Harman (eds.), TREC: Experiment and Evaluation in Information Retrieval (2005); K. Järvelin & J. Kekäläinen, “Cumulated gain-based evaluation of IR techniques”, TOIS 2002; C. Burges et al., “Learning to rank using gradient descent”, ICML 2005; O. Chapelle, D. Metzler, Y. Zhang & P. Grinspan, “Expected reciprocal rank for graded relevance”, CIKM 2009; M. Smucker, J. Allan & B. Carterette, “A comparison of statistical significance tests for information retrieval evaluation”, CIKM 2007; T. Sakai, “Evaluating evaluation metrics based on the bootstrap”, SIGIR 2006; F. Radlinski, M. Kurup & T. Joachims, “How does clickthrough data reflect retrieval quality?”, CIKM 2008; K. Hofmann, S. Whiteson & M. de Rijke, “A probabilistic method for inferring preferences from clicks”, CIKM 2011; O. Chapelle, T. Joachims, F. Radlinski & Y. Yue, “Large-scale validation and analysis of interleaved search evaluation”, TOIS 2012.

Related: Statistics (t-tests and distributions in general), Probability, full-text search in Postgres, and the other search and relevance tools.

Linking to an example

A link opens the tool with the given settings: /tools/ranking-metrics/?<parameters>#<tab>, where the tab is data, metrics, explain, compare or interleave. Share link (top right) builds one from what is on screen; long judgments and runs are packed into ?s=.

Parameter Meaning Example

Parameter	Meaning	Example
`example`	a preset (list below); other parameters in the same link override it	`pooled-trec`
`qrels`	judgments text (TREC or CSV; `%0A` between lines)	`1 0 a 2%0A1 0 b 0`
`a`, `b`	run A and run B text	`1 Q0 a 1 2.5 x`
`na`, `nb`	names of runs A and B	`bm25`
`k`	cut-offs, comma separated	`5,10,20`
`rl`	grade that counts as relevant (trec_eval `-l`)	`2`
`gmax`	maximum grade for ERR	`4`
`M`	depth: only the top M results (0 = all)	`100`
`c`	average over every judged query (`1`/`true`)	`1`
`J`	judged documents only (`1`/`true`)	`1`
`metric`	measure compared: `map`, `recip_rank`, `Rprec`, `P_10`, `recall_10`, `ndcg_cut_10`, `ndcg_exp_10`, `ndcg`, `ndcg_exp`, `err_10`, `err` (k from `k`)	`ndcg_exp_10`
`B`	resamples for randomisation and bootstrap	`10000`
`seed`	random seed of the tests	`1`
`alpha`	significance level	`0.05`
`grades`	One ranking: grades in rank order	`3,2,3,0,1,2`
`extra`	One ranking: grades of relevant documents not retrieved	`3,2`
`ek`	One ranking: cut-off k	`6`
`clicks`	click model: `perfect`, `navigational`, `informational`, `random`	`informational`
`imps`	impressions to simulate	`5000`
`ilen`	interleaved list length	`10`
`iseed`	interleaving seed	`7`

example

a preset (list below); other parameters in the same link override it

pooled-trec

qrels

judgments text (TREC or CSV; %0A between lines)

1 0 a 2%0A1 0 b 0

a, b

run A and run B text

1 Q0 a 1 2.5 x

na, nb

names of runs A and B

bm25

k

cut-offs, comma separated

5,10,20

rl

grade that counts as relevant (trec_eval -l)

2

gmax

maximum grade for ERR

4

M

depth: only the top M results (0 = all)

100

c

average over every judged query (1/true)

1

J

judged documents only (1/true)

1

metric

measure compared: map, recip_rank, Rprec, P_10, recall_10, ndcg_cut_10, ndcg_exp_10, ndcg, ndcg_exp, err_10, err (k from k)

ndcg_exp_10

B

resamples for randomisation and bootstrap

10000

seed

random seed of the tests

1

alpha

significance level

0.05

grades

One ranking: grades in rank order

3,2,3,0,1,2

extra

One ranking: grades of relevant documents not retrieved

3,2

ek

One ranking: cut-off k

6

clicks

click model: perfect, navigational, informational, random

informational

imps

impressions to simulate

5000

ilen

interleaved list length

10

iseed

interleaving seed

7

Presets for example=:

Name Tab Shows

Name	Tab	Shows
`croft-ap`	metrics	Croft et al. §8.4: one query, two rankings — AP 0.71 against 0.33 (open)
`croft-map`	metrics	Croft et al. §8.4: two queries with AP 0.62 and 0.44, MAP 0.53 (open)
`wiki-dcg`	explain	Grades 3,2,3,0,1,2 plus two unretrieved relevant documents: DCG 6.861, IDCG 8.740, nDCG@6 0.785 (open)
`err-cascade`	explain	One perfect answer on top: ERR is nearly done, nDCG keeps counting (open)
`pooled-trec`	compare	25 pooled queries, BM25 against a reranker, nDCG@10 under all four tests (open)
`ties`	metrics	Equal scores ordered by docno descending, and an unjudged document (open)
`interleave`	interleave	Team-draft interleaving of the two pooled runs with navigational users (open)

croft-ap

metrics

Croft et al. §8.4: one query, two rankings — AP 0.71 against 0.33 (open)

croft-map

metrics

Croft et al. §8.4: two queries with AP 0.62 and 0.44, MAP 0.53 (open)

wiki-dcg

explain

Grades 3,2,3,0,1,2 plus two unretrieved relevant documents: DCG 6.861, IDCG 8.740, nDCG@6 0.785 (open)

err-cascade

explain

One perfect answer on top: ERR is nearly done, nDCG keeps counting (open)

pooled-trec

compare

25 pooled queries, BM25 against a reranker, nDCG@10 under all four tests (open)

ties

metrics

Equal scores ordered by docno descending, and an unjudged document (open)

interleave

interleave

Team-draft interleaving of the two pooled runs with navigational users (open)

For example, ?example=pooled-trec&metric=ndcg_exp_10&B=20000#compare tests the exponential-gain nDCG@10 with 20,000 resamples, and ?grades=0works through a ranking that missed one excellent document.

Works with

How to use it

Limits

Checked against

Linking to an example