A car is travelling along a straight road. You have its odometer reading at every moment. Between 2:00 and 2:30 it covered 30 miles. Its average speed over that half-hour was

That is uncontroversial. Now look at the speedometer. At precisely 2:17:04 the needle points at 63 mph. What does that mean?

Speed is distance divided by time. But at a single instant, no time passes and no distance is covered. The formula gives \(0/0\), which is meaningless. Yet the speedometer is not lying: the car really is going 63 mph "right now". Something about that statement is true, and this chapter is about making it precise.

This was a live philosophical problem for two thousand years (Zeno’s arrow paradox is a version of it). Newton and Leibniz resolved it in the 1660s–1680s, and the resolution is the derivative.

We can’t divide by zero time, but we can divide by small times. Let a ball roll down a ramp so that its distance (metres) after \(t\) seconds is

How fast is it going at exactly \(t = 2\)? Compute the average speed from \(t=2\) to \(t = 2 + \Delta t\) for smaller and smaller \(\Delta t\):

The averages are homing in on 20 m/s. And we can see why with algebra instead of a calculator. For any \(\Delta t \neq 0\):

The table is just this formula evaluated at \(\Delta t = 1, 0.1, 0.01, 0.001\). As \(\Delta t \to 0\), the average speed approaches 20. We define the instantaneous speed at \(t=2\) to be that limit:

Notice the trick that dissolves the paradox. We never divide by zero. We divide by small non-zero numbers, simplify while that is legal, and only then ask what happens as the interval shrinks. That is exactly what limits were built for in Chapter 16.

On a distance–time graph, the average speed between two times is the slope of the straight line joining the two points on the curve. The instantaneous speed is the slope of the line that just touches the curve at one point. This picture works for any function, not just motion, so let us name the pieces.

Take \(f(x) = \tfrac14 x^3 - x + 2\) and the point \(P = (1, f(1)) = (1, 1.25)\). The secant from \(P\) to the point at \(x = 1+h\) has slope Try it: Drag the secant to P.

Evaluate for a few values of \(h\): \(h=2\) gives \(2.25\); \(h=1.2\) gives \(1.01\); \(h=0.6\) gives \(0.29\); \(h=0.25\) gives \(-0.046875\). As \(h \to 0\) the slope tends to \(-0.25\), which is the slope of the tangent at \(P\).

Everything so far collapses into one definition.

Read it piece by piece:

If the limit exists, \(f\) is differentiable at \(a\). If we let \(a\) vary, we get a new function, \(x \mapsto f'(x)\), the derivative of \(f\). Finding it is called differentiation.

Calculus was invented twice, by Newton and by Leibniz, and both notations survive. You will meet all of these, sometimes in the same paper.

Leibniz notation earns its keep because it records which variable you are differentiating with respect to. \(\frac{dV}{dr}\) and \(\frac{dV}{dt}\) are different questions about the same volume \(V\): how fast it changes per unit of radius, versus per second. It also makes the chain rule look like fraction cancellation, which (as we’ll see) is a helpful mnemonic and a dangerous proof.

Whenever a quantity changes, its derivative is "how fast, right now", and the difference quotient is "how fast, on average". Both show up constantly.

So the slope of the parabola \(y = x^2\) at \(x\) is \(2x\): zero at the vertex, 2 at \(x = 1\), \(-6\) at \(x = -3\).

Here the simplification step is "combine the fractions":

The slope is always negative (the hyperbola always falls) and gets steeper near \(x = 0\).

Here the simplification step is the conjugate trick from Chapter 16: multiply top and bottom by \(\sqrt{x+h} + \sqrt{x}\) so that the difference of square roots becomes a difference of squares.

Write each function as a power of \(x\):

Every time: bring the exponent down in front, then reduce the exponent by one. That is the power rule, \(\frac{d}{dx}x^n = nx^{n-1}\), and we have now earned it rather than been handed it. The next section proves it.

First principles works, but try it on \(f(x) = x^7\sin x + \sqrt{1 + x^2}\) and you’ll be at it all afternoon. What we want is a small set of rules, each derived once from the definition, that let us differentiate any formula built from standard pieces by taking it apart. This is exactly how a compiler handles expressions: a handful of rules for the building blocks and the ways of combining them, applied recursively.

The building blocks are powers, exponentials, logs and trig functions. The ways of combining are: multiply by a constant, add, multiply, divide, and compose. We need one rule for each.

Constants. If \(f(x) = c\) then \(f(x+h) - f(x) = c - c = 0\), so \(f'(x) = 0\). A flat line has slope zero.

Constant multiple. If \(g(x) = c\,f(x)\):

Stretching a graph vertically by a factor of \(c\) stretches every slope by \(c\).

Sum. If \(s(x) = f(x) + g(x)\):

Together these say differentiation is linear: \(\frac{d}{dx}(af + bg) = af' + bg'\). That word will mean a great deal in Chapter 27.

For a positive whole number \(n\), the binomial theorem (Chapter 7) expands \((x+h)^n\):

The only facts we need: the first term is \(x^n\), the second is \(nx^{n-1}h\), and every remaining term contains at least \(h^2\). So

We proved it for positive integers. It is in fact true for every real exponent \(n\) (we already checked \(n = -1\) and \(n = \tfrac12\) by hand); the general proof comes for free once we have \(\ln x\), later in this chapter.

Worked example. Differentiate \(p(x) = 4x^5 - 3x^2 + \dfrac{2}{x} - 7\sqrt{x} + 9\). Try it: Differentiate it symbolically.

First rewrite every term as a power of \(x\), because the power rule needs that form:

The problem. Is the derivative of a product the product of the derivatives? Try \(x \cdot x = x^2\): the product of derivatives is \(1 \cdot 1 = 1\), but the true derivative is \(2x\). So no. Try it: Grow the product-rule rectangle.

Intuition: a growing rectangle. Picture \(f(x)\,g(x)\) as the area of a rectangle with sides \(f\) and \(g\). Nudge \(x\) by \(h\). The width grows by \(\Delta f\) and the height by \(\Delta g\). The area gains three pieces: a strip \(g\,\Delta f\) down the side, a strip \(f\,\Delta g\) across the top, and a tiny corner \(\Delta f\,\Delta g\).

Why it works. Divide the change in area by \(h\):

As \(h \to 0\), \(\frac{\Delta f}{h} \to f'\) and \(\frac{\Delta g}{h} \to g'\), while \(\Delta g \to 0\) (because \(g\) is continuous), so the corner term vanishes:

In words: "derivative of the first times the second, plus the first times derivative of the second". Each factor takes its turn changing while the other is held fixed.

Worked example. \(\dfrac{d}{dx}\left(x^2 \sin x\right)\). Let \(f = x^2\), \(g = \sin x\), so \(f' = 2x\) and \(g' = \cos x\) (proved in the next section): Try it: Differentiate it symbolically.

Why it works. Let \(Q = f/g\), so \(f = Qg\). Differentiate both sides with the product rule and solve for \(Q'\):

(This derivation assumes \(Q\) is differentiable; a first-principles proof removes that assumption and gives the same formula.) The traditional mnemonic: "low d-high minus high d-low, over the square of what’s below". The order matters because of the minus sign.

Worked example. \(\dfrac{d}{dx}\,\dfrac{x}{x^2 + 1}\). With \(f = x\), \(f' = 1\), \(g = x^2+1\), \(g' = 2x\): Try it: Differentiate it symbolically.

The derivative is zero at \(x = \pm 1\), which is where the graph turns: the maximum value \(\tfrac12\) and minimum \(-\tfrac12\). Finding turning points this way is the subject of Chapter 18.

The problem. How do we differentiate a composition, a function of a function, such as \(\sin(x^2)\) or \((3x^2 + 1)^5\)? Expanding \((3x^2+1)^5\) is possible but miserable; expanding \(\sqrt{1 + x^2}\) is impossible.

Intuition: gears. Three gears are meshed. Gear B turns 3 times for every turn of gear A; gear C turns 2 times for every turn of gear B. How many times does C turn per turn of A? \(3 \times 2 = 6\). Rates of change through a chain multiply.

Now write the composition as a chain. If \(y = (3x^2 + 1)^5\), introduce the inner quantity \(u = 3x^2 + 1\), so \(y = u^5\). Then \(x\) drives \(u\) at rate \(\frac{du}{dx} = 6x\), and \(u\) drives \(y\) at rate \(\frac{dy}{du} = 5u^4\). So \(x\) drives \(y\) at rate

In Leibniz notation the \(du\)s appear to cancel, which makes the rule easy to remember. In Lagrange notation, for \(y = f(g(x))\):

"Derivative of the outer function, evaluated at the inner function, times the derivative of the inner function."

Why it works. Write \(u = g(x)\) and \(\Delta u = g(x+h) - g(x)\). Provided \(\Delta u \neq 0\) for small \(h\):

As \(h \to 0\), \(\Delta u \to 0\) (since \(g\) is continuous), so the first fraction tends to \(f'(u)\) and the second to \(g'(x)\). The proviso "\(\Delta u \neq 0\)" is a genuine gap: for some functions \(\Delta u\) is zero infinitely often near \(x\). The rigorous proof patches it by replacing the first fraction with a function that equals \(f'(u)\) when \(\Delta u = 0\); the result is unchanged. This is the classic case where the intuitive argument is right in spirit and the careful version is a technicality.

Worked examples. Try it: Differentiate (3x² + 1)⁵.

The chain rule is the one you use without noticing, because it is how rates convert between units.

You will often have a choice of route. Some guidance:

Intuition first. Look at the graph of \(\sin x\) and read off its slope. At \(x=0\) it rises steeply (slope 1). At \(x = \pi/2\), the peak, the slope is 0. At \(x = \pi\) it falls as steeply as it rose (slope \(-1\)). At \(3\pi/2\) it bottoms out (slope 0). Plot those slopes and you get \(1, 0, -1, 0, 1, \dots\): the cosine curve.

Why it works. We need two limits established in Chapter 16 (with \(h\) in radians): Try it: Build cos x from slopes.

and the addition formula \(\sin(x + h) = \sin x\cos h + \cos x\sin h\) from Chapter 13. Then

The same computation with \(\cos(x+h) = \cos x\cos h - \sin x\sin h\) gives

From these two and the quotient rule, \(\tan x = \sin x/\cos x\) gives

The problem. What is the derivative of an exponential \(b^x\)? Start from the definition and use an exponent law:

So the derivative of any exponential is proportional to itself. The constant of proportionality, call it \(M(b) = \lim_{h\to0}(b^h - 1)/h\), is the slope of \(b^x\) at \(x = 0\). Numerically: Try it: Slope of 2ˣ at 0.

For base 2 the slope at 0 is about 0.693; for base 3 it is about 1.099. Somewhere between 2 and 3 is the base whose slope at 0 is exactly 1. That base is \(e = 2.71828\ldots\), and for it

This is arguably the best definition of \(e\): the base for which the exponential function is its own derivative. The compound-interest definition \(e = \lim_{n\to\infty}(1 + 1/n)^n\) from Chapter 12 picks out the same number; showing they agree is a nice exercise in limits.

With the chain rule, \(\frac{d}{dx}e^{kx} = k\,e^{kx}\) and \(\frac{d}{dx}e^{g(x)} = g'(x)\,e^{g(x)}\). For example \(\frac{d}{dx}e^{-x^2/2} = -x\,e^{-x^2/2}\): this is the bell curve of Chapter 30, and its slope formula tells you it rises for \(x < 0\) and falls for \(x > 0\). Try it: Slope of eˣ at 0.

\(\ln x\) is the inverse of \(e^x\): \(y = \ln x\) means exactly \(e^y = x\). We know how to differentiate \(e^y\), so differentiate both sides of \(e^y = x\) with respect to \(x\), treating \(y\) as a function of \(x\) and using the chain rule on the left:

Geometrically: the graph of \(\ln\) is the graph of \(\exp\) reflected in the line \(y = x\). Reflection swaps rise and run, so slopes become reciprocals. Where \(e^y\) has slope \(e^y = x\), \(\ln x\) has slope \(1/x\).

Two dividends follow immediately.

Any base. Since \(b = e^{\ln b}\), we have \(b^x = e^{x\ln b}\), and the chain rule gives

So the mysterious constant \(M(b)\) in the table was \(\ln b\) all along: \(\ln 2 = 0.6931\ldots\), \(\ln 3 = 1.0986\ldots\). Similarly \(\frac{d}{dx}\log_b x = \frac{1}{x\ln b}\).

The power rule for every real exponent. For \(x > 0\), \(x^n = e^{n\ln x}\), so

This closes the gap left earlier: \(\frac{d}{dx}x^\pi = \pi x^{\pi - 1}\) is now proved.

The circle \(x^2 + y^2 = 25\) has a perfectly good tangent at \((3, 4)\). What is its slope? The trouble is that the circle is not the graph of a single function: solving gives \(y = \pm\sqrt{25 - x^2}\), two functions glued together. For curves like \(x^3 + y^3 = 6xy\) there is no convenient formula for \(y\) at all.

We don’t need a formula for \(y\). Near any point where the curve is smooth, it locally looks like the graph of some function \(y(x)\), even if we can’t write it down. So differentiate the equation as it stands, treating \(y\) as an unknown function of \(x\). Each time you differentiate an expression in \(y\), the chain rule produces a factor \(\frac{dy}{dx}\); for example \(\frac{d}{dx}y^2 = 2y\frac{dy}{dx}\). Then solve the resulting equation for \(\frac{dy}{dx}\).

At \((3, 4)\): \(y' = -\tfrac34\). Check against geometry: the radius to \((3,4)\) has slope \(\tfrac43\), and a tangent to a circle is perpendicular to the radius, so its slope is the negative reciprocal, \(-\tfrac34\). Correct. At \((3, -4)\) the same formula gives \(+\tfrac34\), automatically handling the lower half.

Find the slope of \(x^3 + y^3 = 6xy\) at \((3, 3)\). First check the point: \(27 + 27 = 54 = 6\cdot 3\cdot 3\). Good.

At \((3,3)\): \(y' = \dfrac{6 - 9}{9 - 6} = -1\).

Inverse functions are the other big use. \(y = \arctan x\) means \(\tan y = x\) with \(-\tfrac{\pi}{2} < y < \tfrac{\pi}{2}\): Try it: Differentiate arctan x.

A neat surprise: a trig-flavoured function whose derivative is a plain rational function. It is why \(\pi\) appears in \(\int_{-\infty}^{\infty}\frac{dx}{1+x^2} = \pi\).

The derivative of a function is itself a function, so we can differentiate it. Does that mean anything? In mechanics it means a great deal. If \(s(t)\) is position, then \(s'(t)\) is velocity, and the rate of change of velocity is acceleration: the quantity in Newton’s second law, \(F = ma\).

A ball is thrown straight up at 20 m/s from ground level. Taking \(g = 9.8\) m/s², its height after \(t\) seconds is \(s(t) = 20t - 4.9t^2\) metres.

Read the story from the derivatives. Acceleration is constant and negative: gravity, pulling down at all times, including at the top. Velocity starts at 20 m/s and decreases steadily. The ball is highest when it stops rising, \(v(t) = 0\): \(t = 20/9.8 \approx 2.04\) s, at height \(s(2.04) \approx 20.4\) m. Going the other way (from acceleration back to position) is integration, and in the Mechanics course you did it every time you used \(s = ut + \tfrac12 at^2\): that formula is just "integrate constant acceleration twice".

The second derivative has a geometric meaning too: it is the rate at which the slope changes, so it measures concavity. \(f'' > 0\) means slopes are increasing (the graph bends upwards, like a cup); \(f'' < 0\) means it bends downwards. That is the backbone of the second-derivative test in Chapter 18.

A pattern for the curious. Differentiate \(\sin x\) repeatedly: \(\cos x, -\sin x, -\cos x, \sin x, \dots\): period four. Differentiate \(x^4\): \(4x^3, 12x^2, 24x, 24, 0, 0, \dots\): a polynomial of degree \(n\) dies after \(n+1\) differentiations. Both facts are the seeds of Taylor series, previewed in Chapter 18.

Where else second derivatives turn up. A news report saying "inflation is still rising, but more slowly" is a statement about a second derivative: prices are increasing (\(P' > 0\)) and the rate of increase is falling (\(P'' < 0\)). In a physics engine, Newton’s second law \(F = m\ddot{x}\) is turned into code by computing acceleration from forces each frame and stepping velocity and position forward (Euler’s method, Chapter 19). And in optimisation the second derivative measures curvature, telling Newton’s method how big a step to take (Chapter 18).

Every rule so far assumed the limit in the definition exists. It doesn’t always. Understanding how it can fail sharpens what the derivative means, and the failures appear in practice: the ReLU activation in neural networks has a corner, and absolute-value loss functions have corners exactly at their minimum. So does UK income tax: tax owed is a continuous function of income, but its slope (your marginal rate) jumps from 20% to 40% at the higher-rate threshold (£50,270 in recent tax years). Just below it, an extra £100 of pay costs £20 in income tax; just above, £40. The graph has a corner there, and "the" derivative at the threshold does not exist.

A corner: \(f(x) = \lvert x \rvert\) at \(x = 0\). The difference quotient is \(\dfrac{\lvert h \rvert - 0}{h}\), which equals \(+1\) for \(h > 0\) and \(-1\) for \(h < 0\). The one-sided limits disagree (\(+1\) from the right, \(-1\) from the left), so the two-sided limit does not exist. Geometrically: secants from the right and secants from the left settle on two different lines. There is no single tangent.

A cusp: \(f(x) = x^{2/3}\) at \(x = 0\). The difference quotient is \(\dfrac{h^{2/3}}{h} = h^{-1/3}\), which tends to \(-\infty\) as \(h \to 0^-\) and to \(\infty\) (positive infinity) as \(h\) shrinks to 0 from the right. The graph comes into the origin vertically from both sides and turns back on itself.

A vertical tangent: \(f(x) = x^{1/3}\) at \(x = 0\). The difference quotient is \(\dfrac{h^{1/3}}{h} = h^{-2/3} \to +\infty\) from both sides. There is a tangent line (the \(y\)-axis), but its slope is infinite, and the derivative must be a finite number.

A fourth: discontinuity. If \(f\) jumps at \(a\), it cannot be differentiable there. In fact: Try it: Left and right slopes of |x|.

Proof (one line, and worth it): \(f(a+h) - f(a) = \dfrac{f(a+h) - f(a)}{h}\cdot h \to f'(a)\cdot 0 = 0\) as \(h \to 0\), so \(f(a+h) \to f(a)\). The converse is false: \(\lvert x \rvert\) is continuous everywhere but not differentiable at 0.

The definition says the difference quotient approaches \(f'(x)\) as \(h \to 0\). So pick a small \(h\) and compute it. Two common formulas:

The central difference uses a secant that straddles \(x\) symmetrically. Draw it on a parabola and you’ll see why it’s better: the errors on the two sides cancel. In fact for a quadratic the central difference is exact for any \(h\).

How big is the error? Taylor polynomials (Chapter 18) tell us \(f(x+h) = f(x) + hf'(x) + \tfrac{h^2}{2}f''(x) + \tfrac{h^3}{6}f'''(x) + \cdots\). Substituting:

This truncation error says: make \(h\) smaller, get a better answer. Shrinking \(h\) by 10 cuts the forward error by 10 and the central error by 100.

But computers don’t do real arithmetic; they do floating point (Chapter 1), with about 16 significant digits (\(\varepsilon \approx 2.2\times 10^{-16}\)). When \(h\) is tiny, \(f(x+h)\) and \(f(x)\) agree in most of their digits, and subtracting them (catastrophic cancellation) leaves only a few meaningful digits. Each value carries a rounding error of about \(\varepsilon\lvert f \rvert\), and dividing by \(h\) amplifies it to about \(\varepsilon\lvert f \rvert / h\). This rounding error grows as \(h\) shrinks.

Let’s measure both. Differentiate \(\sin\) at \(x = 1\); the exact answer is \(\cos 1\).

The theory predicts the top of the table exactly: forward error \(\approx \tfrac{h}{2}\lvert\sin 1\rvert = 0.42h\), central error \(\approx \tfrac{h^2}{6}\cos 1 = 0.090h^2\). Then the trend reverses. The forward difference is best around \(h \approx 10^{-8} \approx \sqrt{\varepsilon}\), with about 8 correct digits; the central difference is best around \(h \approx 10^{-5} \approx \varepsilon^{1/3}\), with about 11. Make \(h\) smaller than that and the answer gets worse.

Numerical differentiation is approximate and fragile. Symbolic differentiation (applying the rules to a formula tree, as SymPy does) is exact but needs a formula and can produce enormous expressions. There is a third way, and it powers essentially all modern machine learning: automatic differentiation (AD). It applies the rules of this chapter, not to formulas, but to the actual numbers flowing through your program. Try it: Dual numbers, node by node.

The forward-mode version rests on a small algebraic trick. Invent a number \(\epsilon\) with \(\epsilon \neq 0\) but \(\epsilon^2 = 0\) (like \(i^2 = -1\) in Chapter 15, it’s an extension of the number system defined by a rule). Numbers of the form \(a + b\epsilon\) are called dual numbers. Multiply two:

The \(\epsilon\) part is the product rule: if \(b\) and \(d\) are the derivatives of two quantities \(a\) and \(c\), then \(ad + bc\) is the derivative of their product. More generally, for any polynomial \(f\), expanding \(f(x + \epsilon)\) gives \(f(x) + f'(x)\epsilon\) exactly, because every \(\epsilon^2\) and higher term vanishes. Evaluate your function on \(x + 1\epsilon\) and the derivative falls out in the \(\epsilon\) slot, exact to floating-point precision, with no \(h\) to choose.

Look at what each method of the class contains: the sum rule, the product rule, the quotient rule, and (in sin and exp) the chain rule. The rules of this chapter are the whole implementation. The derivatives agree with the hand-derived formulas to every printed digit.

Forward mode pushes one derivative through the program alongside the values. That costs one pass per input variable. A neural network has millions of inputs (its weights) and one output (the loss), so forward mode would need millions of passes. Reverse mode turns it around: run the program forward once, recording intermediate values, then walk backwards multiplying local derivatives via the chain rule. One backward pass gives the derivative of the output with respect to every input. In machine learning, reverse-mode AD is called backpropagation. Try it: Backpropagate the one-weight network.

Here is a one-weight "network": input \(x\), weight \(w\), prediction \(a = \tanh(wx)\), loss \(L = (a - \text{target})^2\). As a chain: \(w \to z = wx \to a = \tanh z \to L\). The chain rule says

using \(\frac{d}{dz}\tanh z = 1 - \tanh^2 z\).

The central difference is used here the way practitioners use it: as a gradient check, an independent test that the hand-written backward pass is right. A real network has millions of weights and many layers, so the chain has many links and branches, but every step of backpropagation is this: a local derivative, multiplied into the running product, from the output back towards the inputs. Gradient descent (Chapter 18, Chapter 20, Chapter 33) then nudges each weight downhill.