Almost every question of the form "what is the best …​?" is an optimisation problem. What is the best straight line through these data points? What are the best weights for this neural network? What is the cheapest route, the strongest beam for a given mass of steel, the portfolio with the least risk for a target return?

In each case there is a number you want to make as small (or large) as possible, and a set of knobs you are allowed to turn. Machine learning in particular is, at its core, the business of turning millions of knobs to make one number — the loss — as small as possible.

Picture the loss as a landscape. Each point on the ground is a setting of the knobs; the height is the loss. You are standing somewhere on this landscape in thick fog. You cannot see the lowest valley, but you can feel the slope under your feet. The sensible strategy is to step downhill, feel the slope again, and repeat. That is gradient descent, which you met in Chapter 20: the gradient points uphill, so you walk in the opposite direction.

The fog is the crucial part of the picture. Any optimisation method only ever has local information — the value and slope at the current point. Whether local information is enough to find the global answer depends entirely on the shape of the landscape.

Some landscapes are friendly. A bowl has exactly one low point, and from anywhere on its surface "downhill" eventually takes you there. Other landscapes are egg boxes, with many hollows; walking downhill takes you to the nearest hollow, which may not be the deepest.

The mathematical word for "bowl-shaped" is convex.

The key theorem, stated without proof: for a convex function, every local minimum is a global minimum. The proof is short and worth seeing in outline. Suppose \(a\) is a local minimum but some point \(b\) is lower. Walk along the chord from \(a\) towards \(b\). Convexity says the function along that chord is at most the chord’s height, and the chord slopes down from \(f(a)\) to \(f(b) < f(a)\). So arbitrarily close to \(a\) there are points lower than \(a\) — contradicting "local minimum". Hence no such \(b\) exists.

That is why optimisers love convex problems: the fog does not matter. Any method that reliably goes downhill will find the answer.

Linear regression (Section 33.2) and logistic regression (Section 33.3) have convex losses. Neural networks (Section 33.4) do not. One of the surprises of the last decade is that gradient descent still works astonishingly well on neural networks; why it works so well is an active research question.

Recall the basic update from Chapter 20:

Here \(t\) counts steps, \(\eta\) (Greek eta) is the learning rate, and \(\nabla L(\theta_t)\) is the gradient of the loss at the current parameters. Two practical problems motivate two variants.

Problem 1: the gradient is expensive. In machine learning the loss is an average over a data set,

where \(\ell_i\) is the loss on example \(i\). If \(n\) is a billion, computing the exact gradient costs a billion evaluations per step. Stochastic gradient descent (SGD) instead estimates the gradient from a small random mini-batch of, say, 32 examples. The estimate is noisy but unbiased: on average it points the right way. You get a thousand cheap, slightly wobbly steps for the price of one exact step, and that is almost always a better deal.

Problem 2: narrow valleys. If the landscape is a long, narrow ravine, the gradient points mostly across the ravine (where it is steep) rather than along it (where the minimum is). Plain gradient descent zig-zags from wall to wall and crawls along the floor. Momentum fixes this by letting the step accumulate a running "velocity": components of the gradient that keep pointing the same way (along the valley) build up, while components that flip sign every step (across the valley) cancel out.

Take the ravine \(f(x, y) = \tfrac{1}{2}(x^2 + 25y^2)\). Its minimum is at the origin, but it is 25 times steeper in \(y\) than in \(x\). The gradient is Try it: Race momentum against descent.

Start at \((10, 1)\) with \(\eta = 0.07\).

Plain gradient descent, two steps.

The \(y\)-coordinate bounces \(1 \to -0.75 \to 0.5625\): it overshoots each time. The \(x\)-coordinate creeps down by only 7% per step.

Momentum with \(\beta = 0.7\), two steps. The first step is identical because the velocity starts at zero: \(\mathbf{v}_1 = (10, 25)\), position \((9.3, -0.75)\). Then

Look at what happened to the velocity. In \(x\), the old velocity and new gradient agreed, so the velocity grew to 16.3 and the step along the valley was 75% bigger. In \(y\), they disagreed (25 versus \(-18.75\)), so they almost cancelled and the bounce was damped. That is the whole idea of momentum, visible in two steps of arithmetic.

Now implement and run it for 100 steps:

You can predict the first number by hand: in \(x\) each plain step multiplies the coordinate by \(1 - 0.07 = 0.93\), and \(10 \times 0.93^{100} \approx 7.05 \times 10^{-3}\). Momentum is more than four orders of magnitude closer after the same number of gradient evaluations.

A reasonable rule of thumb: closed form if it exists and fits in memory; Newton or quasi-Newton (L-BFGS) for smooth problems with up to a few thousand parameters; SGD with momentum or Adam for huge data and huge models; derivative-free when nothing else applies.

Real problems have rules: probabilities must be non-negative and sum to 1, a robot arm’s joint angles are limited, a budget cannot be exceeded. A constrained problem is written

Three standard ways to handle the "subject to":

You have recorded how many hours five students studied and their test scores (out of 6):

The points do not lie on a line, but there is clearly a trend. What is the best straight line through them? And what does "best" even mean?

This tiny question contains, in miniature, the whole workflow of machine learning: choose a model, choose a loss that says how wrong the model is, and optimise.

A line \(\hat y = b + wx\) makes a prediction \(\hat y_i\) (read "y-hat") for each \(x_i\). The vertical gap \(y_i - \hat y_i\) is the error, or residual. We want the residuals small, all at once. Adding them up fails — positive and negative errors cancel — so we square them first. Squaring makes every error count positively, punishes large errors more than small ones, and (crucially) produces a smooth, convex bowl that calculus can minimise exactly.

The loss is a function of the two parameters:

At the minimum of a smooth bowl, both partial derivatives are zero (Chapter 20). Differentiate using the chain rule; the outer function is "square", the inner is \(y_i - b - wx_i\):

Set each to zero, divide by \(-2\), and split the sums:

These two linear equations in \(b\) and \(w\) are the normal equations. You know how to solve them from Chapter 6.

For the study data, \(n = 5\): Try it: Normal equations, step by step; Fit the least-squares line.

The normal equations are

Multiply the first by 3 and subtract it from the second:

So the best line is \(\hat y = 2.2 + 0.6x\): each extra hour of study is associated with 0.6 more points. The predictions at \(x = 1, \dots, 5\) are 2.8, 3.4, 4.0, 4.6, 5.2; the residuals are \(-0.8, 0.6, 1.0, -0.6, -0.2\) (they sum to zero, as the first normal equation guarantees); and

With many input features, writing out sums becomes hopeless. Matrices (Chapter 27) package it. Stack the parameters into \(\theta = [b,\ w]^{\mathsf T}\), the targets into \(\mathbf{y}\), and the inputs into the design matrix \(X\). Then all predictions at once are \(X\theta\), the residual vector is \(\mathbf{y} - X\theta\), and

That last line is the normal equations in matrix form, valid for any number of features. For the study data,

which are precisely the coefficients we found by hand. Two derivations, one by sums and partial derivatives, one by matrices and gradients, land on the same equations. That is a good sign you understand both.

Instead of solving the equations, we can walk downhill on the MSE. Its gradient is \(\frac{2}{n}X^{\mathsf T}(X\theta - \mathbf{y})\). Starting from \(\theta = \mathbf{0}\) with \(\eta = 0.05\), the first step is Try it: Gradient descent versus the solve.

Both methods in code:

Gradient descent takes about a thousand steps to match what the normal equations gave in one solve. On five points the closed form obviously wins. So why does anybody use gradient descent for regression?

And the stochastic version, on a thousand noisy points from the line \(y = 3x + 2\): Try it: Run SGD yourself.

SGD lands close but not exactly on the answer: with a constant learning rate it keeps jittering around the minimum, as the Watch out box in Section 33.1 warned. Decaying \(\eta\) over the epochs would tighten it.

Try it: Outliers: squared versus absolute.

Now the question is yes/no. Ten students studied for between half an hour and five hours; some passed (1) and some failed (0):

We want a model that outputs the probability of passing. A straight line is wrong for the job: it happily predicts probabilities of \(-0.3\) or 1.4, and a single extreme student drags it around. We need a function that takes any real number and squeezes it into \((0, 1)\).

The sigmoid (or logistic) function is

Read it piece by piece. For large positive \(z\), \(e^{-z}\) is tiny, so \(\sigma(z) \approx 1/1 = 1\). For large negative \(z\), \(e^{-z}\) is huge, so \(\sigma(z) \approx 0\). At \(z = 0\), \(\sigma(0) = 1/(1+1) = 1/2\). It is an S-shaped curve that turns a "score" of any size into a probability.

Logistic regression computes a linear score and passes it through the sigmoid:

What should the loss be? Squared error is tempting but leads to a non-convex problem here. A principled answer comes from probability (Chapter 29): choose \(w, b\) to make the observed data as likely as possible — maximum likelihood.

If the model says student \(i\) passes with probability \(p_i\), then the probability of the observed outcome \(y_i \in \{0, 1\}\) is \(p_i\) if they passed and \(1 - p_i\) if they failed. A neat way to write both cases at once is \(p_i^{y_i}(1-p_i)^{1-y_i}\) (one of the exponents is always zero). Treating students as independent, the likelihood of the whole data set is the product, and

The symbol \(\prod\) (capital Greek pi) is the product analogue of \(\sum\): multiply the terms rather than add them. The final expression is the binary cross-entropy loss. You will see in Section 33.6 that "cross-entropy" is a precise term from information theory, not a coincidence of naming.

The gradient turns out to be beautifully simple. Two ingredients.

Derivative of the sigmoid. Write \(\sigma(z) = (1 + e^{-z})^{-1}\) and use the chain rule:

Derivative of the loss for one example. Let \(\ell = -[y\ln p + (1-y)\ln(1-p)]\) with \(p = \sigma(z)\), \(z = wx + b\). Chain rule, link by link:

The gradient is "prediction minus truth, times input" — exactly the same shape as for linear regression. The messy sigmoid derivative cancelled perfectly against the logarithms. That cancellation is no accident; it is why cross-entropy and sigmoid are always used together.

Try it: The fitted logistic curve.

By hand, the first step: with \(w = b = 0\) every \(p_i = \sigma(0) = 0.5\). Five students passed and five failed, so

and \(b\) does not move yet. For \(w\) we need \(\sum_i x_i = 27.5\) and the sum of the passers' hours, \(2.0 + 3.0 + 4.0 + 4.5 + 5.0 = 18.5\):

The slope starts positive: more hours, higher pass probability. The code runs the same update 5,000 times.

Check one prediction by hand: at one hour, \(z = 1.353 - 3.721 = -2.368\), \(e^{2.368} \approx 10.68\), and \(\sigma(-2.368) = 1/(1 + 10.68) \approx 0.086\). The decision boundary is where \(p = 0.5\), that is where \(z = 0\): \(x = -b/w = 3.721/1.353 \approx 2.75\) hours. The slope means each extra hour multiplies the odds of passing by \(e^{1.353} \approx 3.9\).

Logistic regression is the natural first model for binary classification: convex (so training always finds the best fit), fast, and interpretable — each weight is a log-odds ratio you can explain to a colleague. Alternatives include decision trees and gradient-boosted trees (often more accurate on tabular data, less interpretable), support-vector machines, and neural networks, which are, as the next section shows, stacked logistic regressions. With more than two classes, the sigmoid generalises to the softmax (Section 33.9).

Logistic regression draws a single straight boundary. Many real problems need curved, tangled boundaries: recognising a handwritten 7, deciding whether a sentence is sarcastic. We want a model flexible enough to learn any reasonable function from data, yet still trainable by gradient descent.

An artificial neuron is exactly logistic regression with several inputs:

It takes a weighted sum of its inputs (a dot product, Chapter 26), adds a bias, and applies a nonlinear activation function. A layer is many neurons side by side, which in matrix form is \(\mathbf{a} = \sigma(W\mathbf{x} + \mathbf{b})\) with \(\sigma\) applied to each component. A network is layers stacked, each feeding the next:

That is all a neural network is: a composition of simple functions (Chapter 11), with a very large number of adjustable parameters.

Why not drop the activation and just stack linear layers? Because the composition of linear maps is linear: Try it: Train without an activation.

A hundred linear layers collapse into one matrix (Chapter 27), so the "deep" network would be no more expressive than linear regression. The nonlinearity between layers is what lets each layer bend the space the next one sees. The universal approximation theorem makes this precise: one hidden layer with enough sigmoid (or ReLU) neurons can approximate any continuous function on a bounded region as closely as you like.

To train with gradient descent, we need \(\partial L / \partial w\) for every weight. The loss is a long composition, and the chain rule (Chapter 17, and its multivariable form in Chapter 20) says: the derivative of a composition is the product of the derivatives of the links. Backpropagation is nothing more than evaluating that product efficiently, starting from the loss and working backwards, reusing each intermediate result.

One neuron, squared-error loss: input \(x = 2\), target \(y = 1\), weight \(w = 0.5\), bias \(b = -1\). Try it: Backprop through one neuron.

Forward pass.

Backward pass, outermost link first:

Both gradients are negative, so gradient descent will increase \(w\) and \(b\), pushing \(a\) up towards the target 1. Now verify against a finite difference (Chapter 32) — the standard "gradient check" every ML engineer learns to do:

In a real network the same pattern repeats through every layer. The vector form of each link is a Jacobian matrix (Chapter 20), and the backward pass is a chain of matrix–vector products, which is why GPUs — machines built for matrix multiplication — train networks so well.

You could apply the chain rule from the inputs forwards instead. For a function with one input and many outputs that would be efficient. But a loss has one output and millions of inputs (the parameters). Going backwards, one pass gives the derivative with respect to all of them at once; going forwards would need one pass per parameter. This is reverse-mode automatic differentiation, and the asymmetry is the entire reason deep learning is computationally feasible.

You type "cat on mat" into a search box. Out of millions of documents, which should come first? Computers cannot compare meanings directly; they compare numbers. So the plan is: turn every document and every query into a vector, and measure how well they match with a single arithmetic operation. That operation is the dot product, and this is the \(\mathbf{q}\cdot\mathbf{d}\) you will meet in every information-retrieval paper.

Give every possible word its own axis. A document becomes a point (a vector) in this enormous space, with coordinate \(j\) saying how important word \(j\) is to the document. Documents about similar things point in similar directions. The query is a vector too. "How relevant is document \(\mathbf{d}\) to query \(\mathbf{q}\)?" becomes "how closely do the two arrows point the same way?"

Recall from Chapter 26 that

so the dot product mixes two things: the angle between the vectors (what we care about — topic) and their lengths (often an accident of document length). Dividing the lengths out gives cosine similarity:

It is 1 for vectors pointing the same way, 0 for perpendicular (no shared words), and \(-1\) for opposite directions (impossible for word counts, which are non-negative, but common for learned embeddings).

Raw counts give too much weight to words like "the". Two adjustments fix this:

The weight of term \(t\) in document \(d\) is \(\operatorname{tf}(t,d)\times\operatorname{idf}(t)\). The logarithm (Chapter 12) is there to compress the scale: a word in 1 document out of a million should count for more than one in 1,000, but not a thousand times more.

Three documents and a query: Try it: TF-IDF for “cat on mat”.

With \(N = 3\): "cat" and "mat" appear only in d1, so \(\operatorname{idf} = \ln 3 \approx 1.0986\). "on", "the" and "sat" appear in two documents: \(\operatorname{idf} = \ln(3/2) \approx 0.4055\). The query vector has non-zero entries only for cat, on and mat, so only those coordinates of each document matter for the dot product:

d1 wins decisively. Note that "the" (twice in d1) contributes nothing to the dot product because the query does not contain it — but it does lengthen \(\mathbf{d}_1\), which the cosine will account for. The code computes both scores:

BM25, the default ranking function in search engines such as Lucene, Elasticsearch and OpenSearch, refines TF-IDF in two ways. Its score is Try it: Tune BM25’s k₁ and b.

Read it piece by piece: for each query term, an IDF weight times a saturating function of the term count \(f(t,d)\). The fraction approaches \(k_1 + 1\) as the count grows, so the tenth mention of "cat" adds much less than the first. The bracket in the denominator normalises for document length \(\lvert d\rvert\) relative to the average length "avgdl": \(b = 0\) ignores length, \(b = 1\) normalises fully. Typical settings are \(k_1 \approx 1.2\) and \(b = 0.75\).

Dense embeddings replace the hand-built word axes with a few hundred learned dimensions. A neural network (Section 33.4) is trained so that texts with similar meanings get vectors with high cosine similarity — so "feline resting on a rug" lands near "cat on mat" despite sharing no words. The retrieval step is unchanged: \(\mathbf{q}\cdot\mathbf{d}\), usually on unit vectors.

Choosing. BM25 is fast, needs no training, and is excellent for exact keyword matches (error codes, names, part numbers). Dense embeddings capture meaning and synonyms but can miss rare exact terms and need a model. Modern systems often run both and combine the scores ("hybrid search").

How much information is in a message? A file compressor has to answer this concretely: how few bits can it get away with? A language model is graded by how well it predicts the next word — but what number measures "well"? And the logistic-regression loss was called cross-entropy. Why?

All three questions have one answer, due to Claude Shannon in 1948.

Learning that something likely happened tells you little; learning that something unlikely happened tells you a lot. "The sun rose this morning" carries almost no information. "It snowed in July in Miami" carries a great deal.

Think of the game of twenty questions. If a number is chosen uniformly from 1 to 8, you can always find it with three yes/no questions ("Is it in the top half?"), because each answer halves the possibilities: \(8 \to 4 \to 2 \to 1\). An event of probability \(1/8\) needs 3 bits to pin down; \(3 = \log_2 8 = -\log_2(1/8)\).

That suggests defining the surprise (or self-information) of an event with probability \(p\) as

Certain events (\(p = 1\)) have surprise 0. Each halving of probability adds one bit. And because logs turn products into sums (Chapter 12), the surprise of two independent events is the sum of their surprises — exactly what "amount of information" ought to do.

A random source — a coin, a die, the next letter of English text — produces outcomes with probabilities \(p_1, \dots, p_k\). Its entropy is the expected surprise (expectation from Chapter 29):

(By convention \(0\log 0 = 0\): impossible outcomes contribute nothing.)

Fair coin. \(H = -\big(\tfrac12\log_2\tfrac12 + \tfrac12\log_2\tfrac12\big) = -\big(\tfrac12(-1) + \tfrac12(-1)\big) = 1\) bit. Try it: Entropy and its Huffman code.

Biased coin with \(P(\text{heads}) = 0.9\):

The biased coin is more predictable, so it carries less than half a bit per flip.

Fair die. Six outcomes of probability \(1/6\): \(H = -6 \times \tfrac16\log_2\tfrac16 = \log_2 6 \approx 2.585\) bits. In general a uniform distribution over \(k\) outcomes has entropy \(\log_2 k\), the maximum possible for \(k\) outcomes.

A skewed four-symbol source with probabilities \(\tfrac12, \tfrac14, \tfrac18, \tfrac18\):

Here is the payoff. For the four-symbol source, use the code

No codeword is a prefix of another, so a stream like 0101100 decodes unambiguously as A, B, C, A. The average length is \(\tfrac12(1) + \tfrac14(2) + \tfrac18(3) + \tfrac18(3) = 1.75\) bits per symbol — exactly the entropy. A naive fixed-length code needs 2 bits per symbol.

That is no coincidence. Each symbol got a codeword of length \(-\log_2 p_i\), its surprise. Shannon’s source coding theorem says that no lossless code can average fewer than \(H\) bits per symbol, and that codes can get arbitrarily close to \(H\). Entropy is the compression limit. Huffman coding (a greedy algorithm on a binary tree, Chapter 25) and arithmetic coding are practical ways to approach it.

The deep moral: to compress well, you must predict well. A compressor is a probability model in disguise. Common symbols get short codes only if you know they are common.

Suppose the true distribution is \(p\) but you designed your code (or your model) for a distribution \(q\). Symbol \(i\) then costs \(-\log_2 q_i\) bits but occurs with frequency \(p_i\), so the average cost is the cross-entropy Try it: KL divergence of two coins.

The extra cost over the best possible is the Kullback–Leibler divergence:

It is always \(\ge 0\), and zero only when \(q = p\): you can never beat the truth. Read \(D_{\mathrm{KL}}(p \,\Vert\, q)\) as "the number of extra bits per symbol you waste by believing \(q\) when the truth is \(p\)".

Now the logistic-regression loss makes sense. For one example, the "true distribution" puts probability 1 on the observed label \(y\), and the model’s distribution is \((p, 1-p)\). Their cross-entropy is \(-[y \ln p + (1-y)\ln(1-p)]\) — exactly the loss we derived from maximum likelihood. Minimising cross-entropy, maximising likelihood, and minimising KL divergence to the data are the same thing. Large language models are trained with exactly this loss on the next token, and their "perplexity" is just \(2^{H(p,q)}\) (or \(e^{H}\) in nats).

The last two lines show the asymmetry of KL divergence. By hand: \(D(\text{biased}\,\Vert\,\text{fair}) = 0.9\log_2\frac{0.9}{0.5} + 0.1\log_2\frac{0.1}{0.5} = 0.9(0.848) + 0.1(-2.322) = 0.763 - 0.232 = 0.531\). The reverse direction is larger, because believing the coin is 90/10 when it is fair makes you very surprised by every tails.

In words: "the gradient of \(f\) at \(\mathbf{x}\)" is the vector of partial derivatives. It points in the direction of steepest increase of \(f\), its length is that steepest rate, and it is perpendicular to the contour through \(\mathbf{x}\). Optimisation algorithms step in the direction \(-\nabla f(\mathbf{x})\) to decrease \(f\) — which is why papers write update rules like \(\theta_{t+1} = \theta_t - \eta\nabla_\theta L(\theta_t)\), where the subscript on \(\nabla_\theta\) says "with respect to the parameters \(\theta\), not the data". Concretely: for \(f(x, y) = x^2 + 3y\), \(\nabla f = (2x, 3)\), and at \((1, 2)\) it is \((2, 3)\). Code: by hand, or loss.backward() and read .grad.

In words: multiply matching components and add. Geometrically it is \(\lVert\mathbf{q}\rVert\lVert\mathbf{d}\rVert\cos\theta\); if the paper says vectors are normalised, it is the cosine similarity. Ranking all documents means computing \(D\mathbf{q}\) for a matrix \(D\) whose rows are documents, then sorting. Code: scores = D @ q; top = np.argsort(-scores)[:10].

In words: conditioning on \(B\) means shrinking the universe to the outcomes where \(B\) happened and re-normalising, which is why we divide by \(P(B)\). Bayes' theorem follows from writing \(P(A \cap B)\) two ways: \(P(A \mid B)P(B) = P(B \mid A)P(A)\). Concretely: a disease with prevalence 1% and a test that is 99% sensitive with a 5% false-positive rate gives \(P(\text{ill} \mid +) = \frac{0.99 \times 0.01}{0.99 \times 0.01 + 0.05 \times 0.99} = \frac{0.0099}{0.0594} \approx 0.167\) — only one positive in six is truly ill (Chapter 29). In ML papers, \(p(y \mid \mathbf{x}; \theta)\) means "the model’s probability of label \(y\) given input \(\mathbf{x}\)", with the semicolon separating the parameters \(\theta\) from the conditioning.

In words: "the estimated parameters are those, among the allowed ones, that minimise the average training loss plus a penalty on large weights." That one line describes ridge regression, regularised logistic regression and (with a neural-network \(f_\theta\)) most of deep learning. The formula says what is computed, not how: the how is the normal equations, gradient descent or Adam (Section 33.1). Code: the training loop.

In words: "the loss is the average surprise of the true label, averaged over all the data the world could give us." We cannot compute an expectation over the whole world, so we estimate it with an average over the training set, \(\frac1n\sum_i -\log p_\theta(y_i \mid \mathbf{x}_i)\), and SGD estimates that with a mini-batch. Recognising "expectation, estimated by an average" is half of reading ML theory. For a discrete random variable, \(\mathbb{E}[X] = \sum_x x\,P(X = x)\); a fair die has \(\mathbb{E}[X] = (1 + 2 + \dots + 6)/6 = 3.5\).

Try it: Softmax with a temperature.

In words: "exponentiate each score and divide by the total." The output is a probability distribution over \(K\) classes, with the largest score getting the largest probability — a "soft" version of picking the maximum. With \(K = 2\) it reduces to the sigmoid: \(\frac{e^{z_1}}{e^{z_1} + e^{z_2}} = \sigma(z_1 - z_2)\). Adding the same constant to every score changes nothing (it cancels top and bottom), which is the key to computing it safely:

Without the subtraction, \(e^{1000}\) overflows to infinity (Chapter 32). By hand for the first case: \(e^{2} \approx 7.389\), \(e^{1} \approx 2.718\), \(e^{0.1} \approx 1.105\), total \(\approx 11.212\), giving \(7.389/11.212 \approx 0.659\), and so on.

Here is the formula at the heart of the transformer architecture behind modern language models. You now have every piece: Try it: Attention, step by step.

In words: "each token looks up every other token by dot-product similarity, and takes a probability-weighted average of what it finds." A search engine, a softmax and a matrix product — three ideas from this book — make up one line of the most influential formula in current computing.