Model evaluation workbench

Synthetic builds a seeded model you can steer:

Your predictions (CSV): paste, upload or drop a file. Layouts (a header row is optional; commas, tabs or semicolons):

Your CSV stays in this browser (it is remembered for next time) and is never put in a share link. Download as CSV saves the current data in the same format, so a synthetic set can be opened in Python.

The 2 × 2 table at the chosen threshold (positive when score ≥ t, scikit-learn’s rule) — or, for a multi-class model, the K × K table of argmax predictions — as counts or normalised by true class (each row becomes recall), by prediction (each column becomes precision) or overall. Below: accuracy, balanced accuracy, precision, recall, specificity, FPR, NPV, F1, Matthews correlation, Cohen’s κ and Youden’s J; for multi-class, per-class precision, recall, F1, support and one-vs-rest AUC with micro (pools all decisions: equals accuracy), macro (plain mean: rare classes count fully) and weighted (by support) averages, and the log loss. Multi-class data also gets a positive class picker: the curve, threshold, imbalance and calibration views then treat that class against the rest.

The ROC curve with its AUC and the current operating point; the precision–recall curve twice: as average precision (the step function Σ ΔRecall × Precision that average_precision_score computes) and as the trapezoid under straight lines between the points (auc(recall, precision)). They differ because precision does not change linearly between two operating points: the achievable curve between them bends below the straight line (Davis & Goadrich, 2006), so the trapezoid is usually optimistic, most visibly with few points or many tied scores (the ties example: AP 0.727, trapezoid 0.748). Report AP. The PR baseline is the prevalence, not 0.5. Below, the cumulative gain chart (share of positives found against share of people contacted, with random and perfect lines), lift (gain ÷ share contacted) and a decile table.

Type a cost matrix (the cost of each outcome per case, negative for a benefit) and, optionally, a deployment prevalence — the rate of positives the model will meet in use, applied as weights. The tool shows both score distributions with the threshold on top (drag on the chart), the expected cost per case at every threshold with the minimum marked, TPR, FPR, precision and F1 against threshold, the thresholds that maximise Youden’s J and F1, and Bayes' rule for calibrated probabilities: flag when p > (C_FP − C_TN) / C_FP − C_TN) + (C_FN − C_TP, shifted by the prior odds ratio when the prevalence changes (Elkan, 2001). A big gap between the data’s optimum and Bayes' threshold is a sign the scores are not calibrated.

The reliability diagram (fraction positive against mean score in each bin, equal-width or equal-count bins) with the number of cases per bin; the expected and maximum calibration error, the Brier score with Murphy’s decomposition (reliability − resolution + uncertainty) and the log loss. Two recalibrations are fitted on alternate rows and scored on the others: Platt scaling (a logistic fit σ(a·x + b) by Newton’s method, on the raw score as Platt (1999) did or on its logit — a < 1 shrinks over-confident scores) and isotonic regression (the best non-decreasing step function, by pool-adjacent-violators; Zadrozny & Elkan, 2002). Both keep the ranking, so the AUC barely moves. Multi-class models are checked on their top label: confidence is the largest probability, correct means it was the right class (Guo et al., 2017).

Multiply the positives and the negatives (as exact weights — the expected result of over- or undersampling, without the noise) and compare the original and resampled numbers side by side. TPR, FPR, the ROC curve and its AUC do not move; precision, F1, accuracy, the PR curve and average precision all do. Buttons undersample or oversample to 50/50.

Needs a group column. Per group: size, base rate, threshold, selection rate, TPR, FPR, PPV, accuracy and AUC, and the gaps that fairness definitions compare — equal opportunity (TPR; Hardt, Price & Srebro, 2016), equalised odds (TPR and FPR), predictive parity (PPV), demographic parity (selection rate) and the disparate-impact ratio with the four-fifths rule. Per-group thresholds picks each group’s threshold so all reach the overall TPR, and shows what that costs elsewhere. When base rates differ these criteria cannot all hold at once (Chouldechova, 2017; Kleinberg, Mullainathan & Raghavan, 2016).

MAE, MSE, RMSE, R², explained variance, MAPE (with a warning when actual values are zero), median and maximum absolute error and the mean residual; predicted against actual with the y = x line, residuals against predictions with a binned mean (curves, funnels and offsets show up here), and the residual histogram.

Related: Test accuracy and risk (the binormal ROC and predictive values against prevalence, for medical tests), Statistics, Probability, The maths of machine learning (logistic regression and cross-entropy), Neural network designer, Embedding explorer and Classic ML, step by step.

Sources: scikit-learn’s metrics and calibration documentation (model evaluation, probability calibration); T. Fawcett, “An introduction to ROC analysis”, Pattern Recognition Letters 27 (2006); J. Davis and M. Goadrich, “The relationship between precision-recall and ROC curves”, ICML 2006; C. Elkan, “The foundations of cost-sensitive learning”, IJCAI 2001; J. Platt, “Probabilistic outputs for support vector machines”, 1999; B. Zadrozny and C. Elkan, KDD 2002; C. Guo et al., “On calibration of modern neural networks”, ICML 2017; M. Hardt, E. Price and N. Srebro, “Equality of opportunity in supervised learning”, NeurIPS 2016; A. Chouldechova, “Fair prediction with disparate impact”, Big Data 5 (2017).

A link opens the tool with the given settings: /tools/model-eval/?<parameters>#<tab>, where the tab is data, confusion, curves, threshold, calibration, imbalance, fairness or regression. Share link (top right) builds one from whatever is on screen. Links carry the synthetic-data settings and the evaluation settings, never pasted CSV data.

Presets for example=:

For example, this link opens the fraud example, and this one an over-confident weak model on rare positives.