Classic machine learning, step by step

Each classifier tab has the same Data panel: a dataset (two moons, circles, blobs, XOR, a spiral or a linear split), the number of points, the noise, and New sample for another random draw (the seed). 30 % of the points are held out as a test set: filled points train the model, hollow points only test it, so every tab shows a train and a test accuracy and overfitting is something you can see. The plot’s mode:

Shading shows the predicted class and how sure the model is; the line is the decision boundary, where p = ½.

CART as scikit-learn grows it: at each node, try every threshold halfway between neighbouring values of each feature, score each by the drop in impurity (Gini, or entropy in bits), keep the best, and stop at the maximum depth, below min samples to split, when a child would get fewer than min samples per leaf, or when the node is pure. Next ▶ makes one split at a time (breadth first), ◀ undoes one, Play animates the growth and ⏭ Grow finishes it. The dashed box on the plot is the node about to be split, with its chosen line; the chart under it plots every candidate threshold for x₁ and x₂ against its impurity decrease, with the winner ringed, and the diagram draws the tree so far: the split, impurity, size and class counts of each node (up to six levels). The XOR example shows the greedy search’s blind spot: no single split lowers the impurity much, but two levels solve it.

Many deep trees, each grown on a bootstrap sample (n points drawn with replacement, so about 63 % distinct) and choosing each split from a random subset of the features — with two features, scikit-learn’s default √p means one at random per split. The forest averages the trees' class probabilities: the main plot is that vote share. Small multiples show the first eight trees with their own samples (big dots were drawn several times, faint ones are out of bag); tap one, or use Show, to see it large. The out-of-bag error judges every training point only by the trees that never saw it — a free estimate of the test error.

Friedman’s gradient boosting for log loss, as scikit-learn’s GradientBoostingClassifier: the score starts at the log-odds of the class share, F₀ = log(p/(1 − p)); each round fits a small regression tree to the pseudo-residuals y − p and gives each leaf the Newton step Σr / Σp(1 − p), added after shrinking by the learning rate. Step through the rounds: point size shows how wrong the model still is at each training point, the small plot shows the tree added that round (red pushes towards class 1, blue towards 0), and the chart shows train and test log loss by round, with the best test round ringed — more rounds keep lowering the training loss long after the test loss has turned.

Compare draws all seven models, each with its own tab’s settings, on the current data, with their train and test accuracies. Tap one to open its tab.

Seven optimisers race down a loss surface from the same start. Tap the plot to move the start.

The paths animate together (Play, Restart, speed, or drag the Step slider); the chart plots f − f_min on a log scale for each, and the table gives each optimiser’s loss and distance to the nearest minimum at the current step.

⛶ Full screen (or the F key) fills the screen with the tool: the plot grows to fill the space and the side panel can be hidden with Hide panel so the plot goes edge to edge. Escape (or the button) leaves. Where the browser has no full-screen mode (iPhone) the tool covers the window instead.

Tested against scikit-learn 1.9.1 on a fixed 40-point dataset: the decision tree’s root split (feature, threshold, impurity), all node impurities, probabilities and feature importances for Gini and entropy; KNeighborsClassifier probabilities for four settings; GaussianNB means, variances, ε and probabilities; LogisticRegression coefficients (raw and degree-2 features); SVC(kernel="linear") w, b, support-vector counts and accuracy for C = 0.1, 1, 10; and GradientBoostingClassifier probabilities and training deviance. The optimiser steps are checked against the update formulas in the PyTorch documentation and the schedules against their definitions.

Sources: L. Breiman, J. Friedman, R. Olshen, C. Stone, Classification and Regression Trees (1984); L. Breiman, Random forests (2001); J. H. Friedman, Greedy function approximation: a gradient boosting machine (2001); J. Platt, Sequential minimal optimization (1998) and R.-E. Fan, P.-H. Chen, C.-J. Lin, Working set selection using second order information (2005); S. Shalev-Shwartz et al., Pegasos (2007), for the primal view of the same hinge loss; Y. Nesterov (1983) and I. Sutskever et al. (2013) for momentum; J. Duchi, E. Hazan, Y. Singer, AdaGrad (2011); G. Hinton, Lecture 6e of Neural Networks for Machine Learning (RMSProp, 2012); D. Kingma, J. Ba, Adam (2015); I. Loshchilov, F. Hutter, Decoupled weight decay regularization (AdamW, 2019) and SGDR (cosine annealing, 2017); L. Smith, Super-convergence (one-cycle, 2018); scikit-learn’s user guide and the torch.optim documentation.

Related: Neural network designer (the next step: train a network on the same kinds of data), The maths of machine learning (least squares, gradient descent, logistic regression in one dimension, backpropagation), Model evaluation (confusion matrices, ROC and PR curves, calibration), Embedding explorer, PCA and SVD and Statistics.

A link opens the tool with the given settings: /tools/ml-visualiser/?<parameters>#<tab>, where the tab is tree, forest, boost, knn, logreg, svm, bayes, compare or optim. Share link (top right) builds one from whatever is on screen; points you added or erased are packed into the link too.

Presets for example=:

For example, https://geekychris.github.io/tools/ml-visualiser/?example=tree-xor#tree opens the XOR tree, and https://geekychris.github.io/tools/ml-visualiser/?surf=quadratic&kappa=80&on=sgd,momentum&steps=300#optim an even worse-conditioned bowl.