A hiker stands on a hillside holding a map. The map shows position as two numbers: east–west (\(x\)) and north–south (\(y\)). At every position the ground has a height. So height is a function of two inputs, \(h(x, y)\).

The hiker’s questions are calculus questions. How steep is it here? If I walk north-east, will I climb or descend, and how quickly? Which direction climbs fastest? Where is the summit? With one input there was only one direction to move in, forwards or backwards along the \(x\)-axis. With two inputs there are infinitely many directions. The single-variable derivative has to be extended to handle that.

A function of two variables takes a point \((x, y)\) in the plane and returns a number. You can picture that in three ways, and you should be able to move between them:

Contour plots are often more useful than surfaces. A surface hides its back side and is hard to read numerically. A contour plot shows everything at once, and the spacing of the contours tells you about steepness: closely packed contours mean steep ground, widely spaced contours mean gentle ground. Hold on to that sentence. It comes back when we meet the gradient.

Take \(f(x, y) = x^2 + 3y^2\). What do its level curves look like? Try it: See the bowl’s elliptical level curves.

Set \(f(x, y) = c\) for a few values of \(c\):

Each level curve with \(c > 0\) is an ellipse. Along the \(x\)-axis, the \(c = 3\) ellipse reaches \(x = \pm\sqrt{3} \approx \pm 1.73\). Along the \(y\)-axis it reaches only \(y = \pm 1\). The ellipses are wider than they are tall, so the bowl rises more steeply in the \(y\) direction than in the \(x\) direction. That matches the formula: the coefficient 3 on \(y^2\) makes \(y\) count for more. Negative values of \(c\) give no level curve at all, because the bowl never goes below zero.

We’ll use this bowl later as the test case for gradient descent.

On the hill \(h(x, y)\), "the slope" is ambiguous. The ground can climb steeply to the north and be level to the east at the same point. The derivative of Chapter 17 asked how the output changes when the single input changes. Now there are two inputs, so we have to ask a more specific question: how does the output change when one input changes and the others stay put?

Here is the key move, and it’s simpler than it sounds. Freeze all inputs but one. Then the function really is a function of one variable, and you already know how to differentiate it.

Geometrically, freezing \(y = 2\) slices the surface with the vertical plane \(y = 2\). The slice is an ordinary curve, and its slope at a point is the partial derivative with respect to \(x\). Freezing \(x\) instead slices in the other direction and gives the partial derivative with respect to \(y\).

The partial derivative is the single-variable derivative applied to a slice, so its definition is the familiar limit with only one input nudged:

Look at what changes in each numerator. In the first, only the first argument moves (\(a \to a + h\)) and \(b\) is untouched. In the second, only \(b\) moves. Nothing else is new. Every rule from Chapter 17 (power, product, quotient, chain) still works. The one new habit is to treat the frozen variables as constants.

Find both partial derivatives of Try it: Partials as slices at (1, 2).

at the point \((1, 2)\).

With respect to \(x\) (freeze \(y\), so \(y^2\) behaves like a number such as 4):

With respect to \(y\) (freeze \(x\)):

At \((1, 2)\):

Interpretation. Standing at \((1, 2)\), where \(f(1,2) = 1 + 8 - 2 = 7\), a small step \(\Delta x\) east changes \(f\) by about \(11\,\Delta x\). A small step \(\Delta y\) north changes it by about \(7\,\Delta y\). These are the slopes of the two dashed tangent lines in the slice figure above.

Second partials (we’ll need them for the Hessian):

The mixed partials agree, as Clairaut promised. At \((1, 2)\): \(f_{xx} = 6\), \(f_{yy} = 4\), \(f_{xy} = 8\).

When you have code for \(f\) but no formula, or you just want to check your algebra, approximate each partial with a central difference. Nudge one input up and down by a small \(h\) and leave the others alone:

Choosing a method. Use the symbolic rules whenever you have a formula. They give exact answers and formulas you can reuse. Use finite differences to check symbolic or hand-coded derivatives, or when \(f\) is a black box. Avoid them as the main tool when there are many inputs: each partial costs two function evaluations, so a function of a million variables would need two million evaluations per gradient. Chapter 32 explains why \(h\) can be neither too large nor too small.

A function of \(n\) variables has \(n\) partial derivatives. Each one describes the slope in one axis direction. Keeping track of them separately is clumsy. More importantly, the hiker doesn’t only want to walk due east or due north. We want one object that captures all the slope information at a point and can answer questions about any direction. That object is the gradient.

The full story of vectors is in Chapter 26. For this chapter you need only the following.

For our running example \(f(x, y) = x^3 + 2xy^2 - y\), we already have both partials:

So far this is just packaging. The payoff is two geometric facts, which the next two sections prove:

There is also a corollary: \(\nabla f\) is perpendicular to the contour line through the point.

The arrows shrink to nothing at the two peaks. At a summit there is no uphill direction, so the gradient is the zero vector. Remember that observation: finding where \(\nabla f = \mathbf{0}\) is how we will find maxima and minima. Try it: Explore the gradient field.

In one variable, the tangent line \(y = f(a) + f'(a)(x - a)\) was the best straight-line approximation to a curve near \(a\). Close to \(a\), the curve and the line are almost the same. What is the multivariable version? Zoom in far enough on a smooth surface and it looks flat, like the ground under your feet on a round Earth. That flat thing is the tangent plane.

Move from \((a, b)\) to \((a + \Delta x,\, b + \Delta y)\). Do it in two legs: first east, then north.

The two changes add:

Written with \(x = a + \Delta x\) and \(y = b + \Delta y\), this is the tangent plane at \((a, b)\):

In vector form, using the gradient and the dot product, it reads almost exactly like the one-variable tangent line:

This is the most important formula in the chapter. The gradient is the thing you dot with a small step to get the change in \(f\). It works unchanged in a million dimensions.

Find the tangent plane to \(f(x, y) = x^3 + 2xy^2 - y\) at \((1, 2)\), and use it to estimate \(f(1.1,\, 1.9)\). Try it: Tangent plane and its error.

The true value is \(f(1.1, 1.9) = 1.331 + 2(1.1)(3.61) - 1.9 = 1.331 + 7.942 - 1.9 = 7.373\). The estimate is off by 0.027, under 0.4%, for a step of 0.1 in each direction.

You measure a rectangle as \(x = 20\) cm by \(y = 10\) cm, and each measurement may be off by up to \(0.1\) cm. How uncertain is the area \(A = xy\)? Try it: Propagate the measurement error.

So \(A = 200 \pm 3\) cm², about 1.5%. Notice that the error in the short side matters more, because it gets multiplied by the long side. Engineers and experimental scientists use this calculation every day under the name propagation of uncertainty.

Our hiker at \((1, 2)\) on \(f(x, y) = x^3 + 2xy^2 - y\) wants to walk in the direction of the unit vector \(\mathbf{u} = (0.6,\, 0.8)\), roughly north-north-east. How fast does height change per unit distance walked? And of all possible directions, which climbs fastest?

Take a small step of length \(h\) in direction \(\mathbf{u}\), so \(\Delta\mathbf{x} = h\mathbf{u}\). Linearisation says:

As \(h \to 0\) the approximation becomes exact (for smooth \(f\)), which gives the directional derivative:

The partial derivatives are special cases. With \(\mathbf{u} = (1, 0)\) (due east), \(D_{\mathbf{u}} f = f_x\). With \(\mathbf{u} = (0, 1)\), \(D_{\mathbf{u}} f = f_y\).

Now use the geometric form of the dot product. Let \(\theta\) be the angle between \(\nabla f\) and \(\mathbf{u}\). Since \(|\mathbf{u}| = 1\):

Read off the consequences. \(|\nabla f|\) is fixed at the point, so only \(\cos\theta\) varies as you turn to face different directions.

That is the whole argument, and it rests on one fact about the dot product. It also explains the contour rule from the start of the chapter: bunched contours mean the height changes a lot over a short distance, so \(|\nabla f|\) is large.

At \((1, 2)\) on \(f(x, y) = x^3 + 2xy^2 - y\), with direction \(\mathbf{v} = (3, 4)\): Try it: Turn the direction dial.

So walking north-north-east climbs at 12.2 units of height per unit of distance. That is close to the best possible 13.04, because \((0.6, 0.8)\) points at about 53° and the gradient at about 32.5°, only 20° apart: \(13.04 \cos 20.6^\circ \approx 12.2\). Walking in direction \((-0.844, -0.537)\) descends fastest. Walking in direction \((-7, 11)/\sqrt{170}\), perpendicular to the gradient, keeps you on the contour \(f = 7\).

This is the literacy goal from the preface. When you meet

in a paper, unpack it in four layers:

If you can say those four sentences without looking, you have the main idea of this chapter.

A drone flies a path \((x(t), y(t))\) through a room where the temperature is \(T(x, y)\). Its sensor reads \(T(x(t), y(t))\), a function of time alone. How fast does the reading change? It changes for two reasons: \(x\) is changing, and \(y\) is changing. We need a way to combine them.

Over a short time \(\Delta t\), the drone moves by \(\Delta x \approx x'(t)\,\Delta t\) and \(\Delta y \approx y'(t)\,\Delta t\). Linearisation says the temperature changes by

Divide by \(\Delta t\) and let \(\Delta t \to 0\):

where \(\mathbf{r}'(t) = (x'(t), y'(t))\) is the drone’s velocity. The single-variable chain rule "multiply the rates" becomes "multiply along each path of dependence, then add over the paths". A tree diagram makes the bookkeeping mechanical.

The same rule handles intermediate variables that depend on several inputs. If \(z = f(x, y)\) with \(x = x(s, t)\) and \(y = y(s, t)\), then

The temperature is \(T(x, y) = x^2 + xy\) and the drone flies around the unit circle, \(x = \cos t\), \(y = \sin t\). Find \(dT/dt\) at \(t = \pi/4\).

At \(t = \pi/4\), \(x = y = \sin(\pi/4) = \cos(\pi/4) = \tfrac{\sqrt2}{2}\):

Check by the alternative method. Substitute first and use one-variable calculus. \(T(t) = \cos^2 t + \cos t \sin t = \cos^2 t + \tfrac12\sin 2t\). Then \(T'(t) = -2\cos t\sin t + \cos 2t = -\sin 2t + \cos 2t\), and \(T'(\pi/4) = -1 + 0 = -1\). The answers agree.

Choosing a method. For a small problem with explicit formulas, substituting first is often quicker. The chain-rule form wins when the functions are only known numerically, when there are many intermediate variables, or when you need the derivatives with respect to many inputs at once. That last case is what neural networks require.

Where you’d use this. Any sensor moving through a field measures a chain-rule derivative. A drone flying at velocity \((5, 3)\) m/s through air where \(\nabla T = (-0.02, 0.01)\) °C per metre sees its temperature reading change at \(\frac{dT}{dt} = T_x\frac{dx}{dt} + T_y\frac{dy}{dt} = (-0.02)(5) + (0.01)(3) = -0.07\) °C per second, about 4 °C a minute, even if the air itself is not changing at all. Weather balloons, ocean gliders and a phone’s barometric altimeter all rely on this split between "the field changed" and "I moved through the field". In economics, the same formula says how a cost index moves when several prices drift at once.

So far the outer function returned a single number. Many maps return several numbers: a robot arm maps joint angles \((\theta_1, \theta_2)\) to a hand position \((x, y)\). A graphics transform maps \((u, v)\) texture coordinates to screen positions. For a map \(\mathbf{F} : \mathbb{R}^n \to \mathbb{R}^m\), put the gradients of the \(m\) output components in the rows of a matrix:

Example: the polar-to-Cartesian map \(x = r\cos\theta,\ y = r\sin\theta\) has Jacobian

The determinant \(r\) is the area-scaling factor. That’s why a small polar rectangle has area \(r\,dr\,d\theta\), a fact we’ll use for double integrals. In robotics, the Jacobian of the arm’s kinematics converts joint speeds into hand velocity. Inverting it ("which joint speeds give the hand velocity I want?") is the core of real-time arm control. The arm is in trouble exactly where \(\det J = 0\), a singular configuration such as a fully outstretched arm, which cannot move its hand any further outward.

In Chapter 18 you found the maxima and minima of \(f(x)\) by solving \(f'(x) = 0\) and then checking the second derivative. What is the multivariable version, and what new possibilities appear when there is more than one direction?

At the top of a smooth hill or the bottom of a smooth bowl, the ground is level in every direction: the tangent plane is horizontal. That means every directional derivative is zero, which happens exactly when \(\nabla f = \mathbf{0}\).

But a level tangent plane does not guarantee a peak or a pit. Think of a horse saddle or a mountain pass. Along the path over the pass you are at a high point. Along the ridge that the pass cuts through you are at a low point. The tangent plane is horizontal, yet the point is neither a maximum nor a minimum. This saddle point has no counterpart worth mentioning in one variable, and it turns out to be extremely common in high dimensions.

Near a critical point the linear term of the approximation vanishes (the gradient is zero), so the shape is decided by the next term, the quadratic one built from the Hessian. For two variables, the type of that quadratic is decided by one number, the discriminant

Roughly: if \(D > 0\), the surface curves the same way in every direction (a bowl or a dome), and the sign of \(f_{xx}\) says which. If \(D < 0\), it curves up in some directions and down in others, which is a saddle. For the general \(n\)-variable picture, the signs of the Hessian’s eigenvalues decide: all positive means a minimum, all negative a maximum, mixed a saddle. That’s one of the reasons Chapter 28 matters.

Classify the critical points of \(f(x, y) = x^3 - 3x + y^2\). Try it: Classify these critical points.

The local minimum value is \(f(1, 0) = 1 - 3 + 0 = -2\). Is it the global minimum? No. Along the \(x\)-axis, \(f(x, 0) = x^3 - 3x \to -\infty\) as \(x \to -\infty\), so the function has no lowest value at all. The second-derivative test is purely local.

Try it: See D = 0 fail.

You want the minimum of a function \(f(\mathbf{x})\): the loss of a model, the energy of a molecule, the cost of a design. You can compute \(f\) and \(\nabla f\) at any point, but solving \(\nabla f = \mathbf{0}\) symbolically is impossible, because there are too many variables and the equations are too tangled. What can you do?

You are on a mountainside in thick fog and want to reach the valley floor. You can’t see the valley, but you can feel the slope under your feet. A sensible strategy: feel which way is steepest downhill, take a step that way, and repeat. The gradient gives you exactly that local information. Steepest descent is the direction \(-\nabla f\), and that gives the iteration

Here \(\mathbf{x}_k\) is the position after \(k\) steps and \(\eta\) (Greek "eta") is the step-size multiplier.

Linearisation tells us how much one step changes \(f\). With \(\Delta\mathbf{x} = -\eta\nabla f\):

Since \(|\nabla f|^2 > 0\) whenever we’re not at a critical point, a small enough positive \(\eta\) is guaranteed to decrease \(f\). The words "small enough" are where all the practical difficulty lives, because the linear approximation is only trustworthy for small steps. Notice also that the step length is \(\eta|\nabla f|\): steps are automatically long on steep ground and short on gentle ground, so the method slows down as it nears a minimum.

Minimise \(f(x, y) = x^2 + 3y^2\) (the elliptical bowl from the first section) starting at \(\mathbf{x}_0 = (3, 1)\) with \(\eta = 0.1\). The gradient is \(\nabla f = (2x,\ 6y)\). Try it: Step down the bowl.

The loss falls 12 → 6.24 → 3.76 → 2.37. Look at the pattern in the coordinates. Each step multiplies \(x\) by 0.8 and \(y\) by 0.4. That’s no accident:

With \(\eta = 0.1\) the factors are \(0.8\) and \(0.4\). The \(y\) coordinate, in the steep direction, collapses quickly, while \(x\), in the shallow direction, shrinks only by 20% per step. That slow direction sets the overall pace.

The multiplier formulas explain every line of this output.

So convergence requires \(|1 - 2\eta| < 1\) and \(|1 - 6\eta| < 1\), i.e. \(0 < \eta < 1/3\). The steepest direction sets the upper limit on the learning rate, and the shallowest direction sets the speed. The 2 and 6 are the eigenvalues of this bowl’s Hessian, \(H = \begin{bmatrix} 2 & 0 \\ 0 & 6 \end{bmatrix}\), and their ratio, the condition number \(6/2 = 3\), measures how elongated the bowl is. Long thin valleys (large condition number) are slow for gradient descent no matter how you tune \(\eta\). Chapter 28 and Chapter 32 develop that idea.

Rule of thumb: few variables and a formula, solve analytically. Tens to thousands of smooth variables, use Newton-type or quasi-Newton methods (BFGS) from a library. Millions of variables, or data arriving in batches, use gradient descent variants.

You have 8 metres of fencing to build a rectangular pen against a long wall. The wall forms one side, so the fencing covers one side of length \(x\) parallel to the wall and two sides of length \(y\). Maximise the area \(A = xy\) subject to \(x + 2y = 8\).

This is optimisation with a constraint. You can’t wander freely over the \((x, y)\) plane, only along the line \(x + 2y = 8\). Real problems are full of constraints: budgets, capacities, physical laws, probabilities that must sum to 1.

Draw the contours of the objective \(f = xy\) and the constraint line. Walk along the line. As you go, you cross contours of higher and higher area, until you reach a point where the line only touches a contour without crossing it. Beyond it, you’d cross back to lower contours. At that touching point the constraint curve and the contour are tangent.

Two curves that are tangent at a point share a tangent line there, so their perpendiculars are parallel. The perpendicular to a contour of \(f\) is \(\nabla f\). The constraint curve is itself a contour of the function \(g(x, y) = x + 2y\), so its perpendicular is \(\nabla g\). Therefore, at the optimum,

Maximise \(f(x, y) = xy\) subject to \(g(x, y) = x + 2y = 8\). Try it: Watch the gradients align.

The ends of the feasible segment (\(x = 0\) or \(y = 0\)) give area 0, so \((4, 2)\) is the maximum. The side along the wall is twice each of the other sides.

Alternative: substitution. From the constraint, \(x = 8 - 2y\), so \(A(y) = (8 - 2y)y = 8y - 2y^2\). Then \(A'(y) = 8 - 4y = 0\) gives \(y = 2\), \(x = 4\). Same answer, with ordinary one-variable calculus.

Checking the meaning of \(\lambda\). With \(c\) metres of fence instead of 8, the same steps give \(\lambda = c/4\), \(x = c/2\), \(y = c/4\), so the best area is \(A^* = c^2/8\). Then \(dA^*/dc = c/4\), which equals 2 at \(c = 8\). That is exactly \(\lambda\). One extra metre of fence buys about 2 extra square metres of pen.

Choosing a method. Substitution is easier when you can solve the constraint for one variable cleanly. Lagrange multipliers win when you can’t (constraints like \(x^2 + xy + y^4 = 1\)), when there are many variables and constraints, when symmetry makes the \(\lambda\) equations easy, or when the value of \(\lambda\) itself is what you want to know.

In Chapter 19, \(\int_a^b f(x)\,dx\) added up infinitely many thin strips to find the area under a curve. Now suppose \(f(x, y)\) is the depth of snow over a rectangular field. How much snow is there? You need the volume under the surface \(z = f(x, y)\) over a region of the plane.

Chop the region into small rectangles of width \(\Delta x\) and depth \(\Delta y\). Over each one, stand a column whose height is the value of \(f\) at some sample point inside it. Each column has volume \(f \cdot \Delta x\,\Delta y\). Add them all up and refine the grid. The limit of these sums is the double integral:

How do you compute it? Add the columns one row at a time. Fix \(x\) and integrate over \(y\). That gives the area of one vertical slice through the solid, a function of \(x\). Then integrate those slice areas over \(x\), exactly as you found volumes by slicing in Chapter 19. This is an iterated integral, and Fubini’s theorem guarantees that for continuous \(f\) on a rectangle, the order doesn’t matter:

Find \(\iint_R (x^2 + y)\,dA\) over \(R = [0, 1] \times [0, 2]\). Try it: Refine the columns.

Order 1: \(y\) inside. Hold \(x\) fixed:

Order 2: \(x\) inside. Hold \(y\) fixed:

Both orders give \(8/3 \approx 2.667\), as Fubini promised. Dividing by the area of \(R\) (which is 2) gives the average value of \(f\) over the region, \(4/3\).

The column picture translates directly into code, as the midpoint rule in two dimensions:

Each tenfold refinement cuts the error by a factor of 100, the \(O(h^2)\) behaviour of the midpoint rule. But in \(d\) dimensions an \(n\)-per-side grid needs \(n^d\) evaluations. That "curse of dimensionality" is why high-dimensional integrals (in physics, finance and Bayesian statistics) are done by Monte Carlo sampling instead, which Chapter 32 covers.