Click models workbench

Log. Either simulate or paste/open a log.

Behaviour. Searches, sessions, CTR by rank, clicks per search, abandonment (searches with no click), the mean rank of the first and last click, time to first click, query reformulations between consecutive searches of a session — specialisation (terms added), generalisation (terms removed), parallel, repeat, new topic — and how often people search again after an abandoned search compared with after a click.

Click models. Fitted on the first 75 % of the log (in order) and evaluated on the rest, as PyClick does:

You get the held-out log-likelihood and perplexity (overall and per rank), the examination curve (PBM γ_r, DCM λ_r, UBM’s γ(rank, previous click) table, DBN’s γ), and per query the estimates for every document beside its CTR and mean rank — and, for a simulated log, the true values, the correlation and RMSE of the estimated attractiveness, and the nDCG@10 of the documents re-ranked by each model. Open in Ranking metrics turns a model’s relevance estimate into graded judgments (cut-offs adjustable) and sends them, with the ranking shown most often and the ranking by CTR as two runs, to the ranking metrics workbench.

Position bias. Propensities p_r (the chance a result at rank r is looked at) estimated two ways — from PBM’s γ_r, and model-free from swap interventions (Joachims, Swaminathan & Schnabel 2017: the click rate of the usual top result when moved to rank k, over its rate at rank 1) — against the truth when known. Then inverse propensity weighting: each click counts 1/p_r (clipped at τ), which gives an unbiased attractiveness estimate under a position-based user, and an unbiased offline estimate of a ranker’s "rank of relevant results" loss. The table shows how counting raw clicks flatters the ranker that produced the log.

Interleaving (simulated logs). Two rankers with different errors; balanced interleaving (Joachims 2002) and team-draft interleaving (Radlinski, Kurup & Joachims 2008) against an A/B test on a click metric, with the simulated users. One run shows wins, ties, a sign test and example impressions; Run sensitivity repeats many experiments at each size and plots how often each method names the better ranker at p < 0.05.

These follow Chuklin, Markov & de Rijke, Click Models for Web Search (Morgan & Claypool, 2015), and their implementation in PyClick: every parameter starts at ½ (one click in two impressions); EM restarts its counts every iteration from the previous values (50 iterations by default); EM values are capped at 1 − 10⁻⁶. Log-likelihood is the mean over test searches of the mean natural log of P(C_r = c_r | clicks above) per rank; perplexity at rank r is 2 to the minus mean log₂ of the full (unconditional) P(C_r = c_r), and the overall perplexity is the mean over ranks — 1 is perfect, 2 is a coin toss.

Two deliberate differences from PyClick: the cascade model’s conditional click probability before the first click is α_r (PyClick uses the unconditional probability) and a non-click after it counts 1 − 10⁻⁶ (PyClick gives it 10⁻⁶); and DBN is fitted by exact EM — the posterior of examination by forward–backward over the ranks — where PyClick approximates the attractiveness update with the unconditional examination probability. On the test log the exact version has the better held-out likelihood and perplexity.

The tests fit all nine models in PyClick (Python 3.11) and here on the same 1,000-search log: GCTR, RCTR, DCTR, PBM, DCM, UBM and SDBN agree to 10⁻¹² in log-likelihood, perplexity per rank, full and conditional click probabilities and parameters; the cascade model agrees in parameters, full probabilities and perplexity. They also check the DBN forward–backward against brute-force enumeration of the hidden states, that EM never lowers the training likelihood, that full click probabilities are the marginals of the conditional ones, that PBM recovers a simulated examination curve and swap experiments the true propensities, that IPS-weighted clicks recover attractiveness where raw CTR does not, the interleaving invariants, and that the judgments sent to the ranking metrics workbench parse there.

Related: Ranking metrics (offline measures with significance tests, and team-draft interleaving of your own runs), Relevance and scoring explainer (BM25 and friends), Postings lab, Statistics.

The log you paste or open is parsed and modelled in this page and never leaves it; settings and the pasted log are kept in this browser’s local storage. Open in Ranking metrics writes the judgments into that tool’s local storage in the same browser. Nothing is loaded from the network.