You leave a campsite and walk 3 km due east, then 4 km due north. How far are you from camp, and in which direction?

You already know how to answer this. Draw the two legs as a right angle, use Pythagoras for the straight-line distance, \(\sqrt{3^2 + 4^2} = 5\) km, and use trigonometry for the direction, \(\tan^{-1}(4/3) \approx 53.1^\circ\) north of east. But notice what you had to keep track of. Neither "3" nor "4" alone described a leg of the walk; each was a distance in a direction. And the answer, too, is a distance in a direction.

Physics is full of such quantities. If you took Mechanics at A-level, you spent a year with them: displacement, velocity, acceleration, force, momentum. All of them need a size and a direction. Two forces of 10 N acting on a box might cancel completely (opposite directions), add to 20 N (same direction), or combine to about 14.1 N (at right angles). The number 10 alone can’t tell you which.

So we need a mathematical object that carries direction as well as size, and rules for combining such objects that match what happens in the world. That object is the vector.

There are two ways to picture a vector, and the whole of linear algebra gets easier once you can switch between them without effort.

The arrow view. A vector is an arrow: it has a length and points in a direction. Crucially, it has no fixed position. The displacement "3 km east, 4 km north" is the same displacement whether you start at the campsite or at the car park. Slide the arrow anywhere without rotating or stretching it and it’s still the same vector.

The list view. Put the arrow’s tail at the origin of a coordinate system. Its tip lands at some point, say \((3, 4)\). Those two numbers record exactly how far the arrow goes in each axis direction, and they determine the arrow completely. So a vector is the list \((3, 4)\).

The arrow view gives you geometry: you can see addition, angles and projections. The list view gives you computation: you can add, multiply and store vectors in an array. The list view also generalises effortlessly. An arrow in 768 dimensions is impossible to draw, but a list of 768 numbers is just an array, and every rule in this chapter applies to it unchanged.

Vector notation varies between textbooks, countries and fields. You’ll meet all of these, and they mean the same thing.

A light aircraft points due north and flies at 200 km/h through the air. A wind blows from the west at 50 km/h, so the air itself is moving east. What is the plane’s actual velocity over the ground?

The plane’s motion is the result of two motions happening at once. Each is a vector, and we need a rule for combining them.

Think about displacements first, because they are the easiest to picture. If you walk along vector \(\mathbf{a}\), then along vector \(\mathbf{b}\), your total displacement is the single arrow from where you started to where you ended. Draw \(\mathbf{b}\) starting at the tip of \(\mathbf{a}\), then draw an arrow from the tail of \(\mathbf{a}\) to the tip of \(\mathbf{b}\). That arrow is \(\mathbf{a} + \mathbf{b}\). This is the tip-to-tail rule, also called the triangle law.

If you do the walks in the other order, \(\mathbf{b}\) first, you trace the other two sides of a parallelogram and end up in the same place. So \(\mathbf{a} + \mathbf{b} = \mathbf{b} + \mathbf{a}\), and the sum is the diagonal of the parallelogram the two vectors span. That’s the parallelogram law, and it’s how physicists add forces acting at a single point.

Look at the x-coordinates in the diagram. Walking along \(\mathbf{a}\) moves you 4 units right; then \(\mathbf{b}\) moves you 1 more. Total: 5. The y-coordinates are independent of this: 1 up, then 3 more, total 4. Horizontal and vertical motions don’t interfere with each other, so you add them separately:

That is the whole rule, in any number of dimensions: add component by component. The arrow picture and the list calculation always agree, which is why both views are legitimate.

Walking along \(\mathbf{v}\) twice is \(\mathbf{v} + \mathbf{v}\), which in components is every entry doubled. We write it \(2\mathbf{v}\). More generally, for any scalar \(c\), the scalar multiple \(c\mathbf{v}\) multiplies every component by \(c\):

Geometrically, \(c\mathbf{v}\) points along the same line as \(\mathbf{v}\) and is \(|c|\) times as long. If \(c > 1\) it stretches, if \(0 < c < 1\) it shrinks, and if \(c < 0\) it also flips to point the opposite way. \(0\mathbf{v}\) is the zero vector \(\mathbf{0} = (0, 0)\), the only vector with no direction.

Subtraction follows: \(\mathbf{a} - \mathbf{b} = \mathbf{a} + (-1)\mathbf{b}\). Its geometric meaning is worth memorising: \(\mathbf{a} - \mathbf{b}\) is the arrow from the tip of \(\mathbf{b}\) to the tip of \(\mathbf{a}\). (Check: start at \(\mathbf{b}\), add \(\mathbf{a} - \mathbf{b}\), arrive at \(\mathbf{a}\).) This is why "the vector from point \(P\) to point \(Q\)" is \(Q - P\).

Set up axes with \(x\) pointing east and \(y\) pointing north, units km/h.

Ground velocity is the sum:

To turn this back into "speed and direction", use Pythagoras and the tangent:

The plane travels slightly faster than its airspeed and drifts 14° off course. A pilot who wants to track due north must point slightly west so that the wind’s push cancels; finding that heading is a nice exercise below.

A ring is held still by three ropes. Two of them pull with forces \(\mathbf{F}_1 = (3, 4)\) N and \(\mathbf{F}_2 = (-5, 2)\) N. What force must the third rope supply?

"Held still" (in equilibrium) means the forces sum to zero, Newton’s first law. So

The third rope pulls with \((2, -6)\) N, which has magnitude \(\sqrt{4 + 36} = \sqrt{40} \approx 6.32\) N, pointing right and steeply down. Drawn tip to tail, the three forces form a closed triangle: you end where you began.

Mechanics problems often give vectors as "a speed and a bearing" rather than as components. You can add those directly using the law of cosines on the triangle, but that’s error-prone and doesn’t scale beyond two vectors. The reliable method is: convert everything to components, add, convert back. A vector of length \(r\) at angle \(\theta\) above the positive x-axis has components

straight from the unit circle (Chapter 13). After adding, recover the length with Pythagoras and the angle with \(\tan^{-1}\) (or better, atan2, which gets the quadrant right). Choose components whenever more than two vectors are involved, or whenever you’ll compute.

A game character moves with velocity \((1, 0)\) when you press the right arrow and \((0, 1)\) when you press up. Press both, and the code adds them: \((1, 1)\). Players soon notice that moving diagonally is faster than moving straight. How much faster, and how do you fix it?

To answer, we need the length of a vector, and a way to keep a direction while setting the length to whatever we want.

In two dimensions, the arrow \((v_1, v_2)\) is the hypotenuse of a right triangle with legs \(|v_1|\) and \(|v_2|\), so its length is \(\sqrt{v_1^2 + v_2^2}\). For \((1, 1)\) that’s \(\sqrt{2} \approx 1.414\): diagonal movement is about 41% faster. That’s the bug.

In three dimensions, apply Pythagoras twice. The vector \((v_1, v_2, v_3)\) has a "shadow" \((v_1, v_2, 0)\) on the floor, of length \(\sqrt{v_1^2 + v_2^2}\). The full vector rises vertically \(v_3\) above the end of that shadow, making a second right triangle:

Nothing stops the pattern there. Each new axis is perpendicular to all the previous ones, so each adds one more squared term. In \(n\) dimensions, define the length the same way:

The distance between two points \(P\) and \(Q\) is the length of the vector between them, \(\|Q - P\|\). In two dimensions this is exactly the distance formula from Chapter 10; now it works in any dimension.

A unit vector has length 1. Dividing any nonzero vector by its own length produces a unit vector pointing the same way:

This is called normalising \(\mathbf{v}\). The hat notation \(\hat{\mathbf{v}}\) ("v-hat") is the usual marker for a unit vector. Normalising throws away the magnitude and keeps the pure direction; you can then multiply by any length you like. The special unit vectors along the axes, \(\mathbf{i} = (1, 0, 0)\), \(\mathbf{j} = (0, 1, 0)\), \(\mathbf{k} = (0, 0, 1)\), are the standard basis vectors, also written \(\mathbf{e}_1, \mathbf{e}_2, \mathbf{e}_3\).

Normalise \(\mathbf{v} = (2, -1, 2)\). Try it: Normalise (2, −1, 2).

And the game fix: before applying movement, normalise the input direction (when it isn’t zero) and multiply by the desired speed \(s\). Diagonal input \((1, 1)\) becomes \(s \cdot (0.707, 0.707)\), whose length is \(s\), the same as straight movement.

You have two moves available: \(\mathbf{a} = (2, 1)\) and \(\mathbf{b} = (1, -1)\), and you may use each any real number of times (including fractions and negatives). Can you reach the point \((7, 1)\)? Can you reach every point?

An expression like \(x\mathbf{a} + y\mathbf{b}\), scaled vectors added together, is called a linear combination of \(\mathbf{a}\) and \(\mathbf{b}\). Think of \(\mathbf{a}\) and \(\mathbf{b}\) as two directions on a strange compass, and \(x\), \(y\) as how far you travel along each. The span of \(\mathbf{a}\) and \(\mathbf{b}\) is the set of everywhere you can reach.

We need \(x(2, 1) + y(1, -1) = (7, 1)\). Comparing components gives a system of equations (Chapter 6): Try it: Reach (7, 1).

Adding the equations eliminates \(y\): \(3x = 8\), so \(x = 8/3\). Then \(y = x - 1 = 5/3\). Check: \(\tfrac{8}{3}(2, 1) + \tfrac{5}{3}(1, -1) = (\tfrac{16}{3} + \tfrac{5}{3},\ \tfrac{8}{3} - \tfrac{5}{3}) = (7, 1)\). ✓

The same elimination works for any target \((p, q)\): \(x = (p + q)/3\), \(y = (p - 2q)/3\). So \(\mathbf{a}\) and \(\mathbf{b}\) span the whole plane.

Notice the deep fact hiding here: asking whether a vector is a linear combination of others is the same as solving a system of linear equations. That equivalence is the beating heart of linear algebra, and Chapter 27 builds on it.

Two very different questions turn out to have the same answer.

A Mechanics question. You drag a sled 5 m across flat snow, pulling on a rope with a 10 N force angled 60° above the horizontal. How much work do you do? Work is "force times distance moved in the direction of the force", so only part of your pull counts.

A data question. Two documents, or two users, or two songs, are represented by vectors of numbers. How similar are they? We’d like a single number that’s large when the vectors point the same way and small or negative when they don’t.

Both questions ask: how much do two vectors agree in direction, weighted by their sizes? The tool that answers it is the dot product.

Multiply matching components and add up:

The result is a scalar, not a vector, which is why it’s also called the scalar product (or inner product). For example,

The second calculation shows an important link: a vector dotted with itself is its length squared, \(\mathbf{v} \cdot \mathbf{v} = \|\mathbf{v}\|^2\). Directly from the definition, the dot product also satisfies

Each holds because it holds for every term \(a_i b_i\) separately.

That’s the mechanical definition. On its own it looks arbitrary: why multiply matching components? The answer is that this mechanical recipe secretly measures an angle.

Put the tails of \(\mathbf{a}\) and \(\mathbf{b}\) together, with angle \(\theta\) between them. The third side of the triangle runs from the tip of \(\mathbf{b}\) to the tip of \(\mathbf{a}\), which is the vector \(\mathbf{a} - \mathbf{b}\).

Now compute the squared length of the third side in two ways.

Using the dot product algebra:

Using the law of cosines with side lengths \(\|\mathbf{a}\|\), \(\|\mathbf{b}\|\), \(\|\mathbf{a} - \mathbf{b}\|\):

The left sides are equal, so the right sides are equal. Cancel \(\|\mathbf{a}\|^2 + \|\mathbf{b}\|^2\) from both and divide by \(-2\):

This is the central fact of the chapter. The "multiply and add" recipe, which a computer does in nanoseconds, equals "length times length times the cosine of the angle between", which is geometry. Two representations, one number.

Since lengths are positive, the sign of \(\mathbf{a}\cdot\mathbf{b}\) is the sign of \(\cos\theta\):

The middle case gets its own word. Vectors with \(\mathbf{a}\cdot\mathbf{b} = 0\) are orthogonal (perpendicular, at right angles). Earlier we found \((3, 4)\cdot(4, -3) = 0\): those two arrows are perpendicular, which you can confirm by sketching them. This is an astonishingly cheap perpendicularity test: no angles, no trigonometry, just multiply and add. In 768 dimensions, where you can’t sketch anything, it’s the only practical test.

Find the angle between \(\mathbf{a} = (1, 2, 2)\) and \(\mathbf{b} = (2, -1, 2)\). Try it: Angle between two 3D vectors.

Work done by a constant force \(\mathbf{F}\) over a displacement \(\mathbf{d}\) is \(W = \mathbf{F}\cdot\mathbf{d}\). With the rope at 60°:

Check with the geometric form: \(\|\mathbf{F}\|\,\|\mathbf{d}\|\cos 60^\circ = 10 \cdot 5 \cdot 0.5 = 25\) J. The vertical part of your pull (8.66 N) does no work because the sled doesn’t move vertically: its dot product with the displacement is zero. That’s the physical meaning of orthogonality, and it’s why carrying a heavy bag across a level room does, by the physicist’s definition, no work on the bag.

A 4 kg box rests on a ramp inclined at 30°. Gravity pulls straight down. How much of that pull acts along the ramp (trying to slide the box) and how much acts into the ramp (pressing it against the surface)?

Equivalently: given a vector \(\mathbf{a}\) and a direction \(\mathbf{b}\), split \(\mathbf{a}\) into a part along \(\mathbf{b}\) and a part perpendicular to \(\mathbf{b}\). Mechanics calls this resolving a force. Linear algebra calls the first part the projection of \(\mathbf{a}\) onto \(\mathbf{b}\).

Shine a light perpendicular to the line of \(\mathbf{b}\). The shadow that \(\mathbf{a}\) casts on that line is its projection. What’s left over, from the tip of the shadow up to the tip of \(\mathbf{a}\), is perpendicular to \(\mathbf{b}\).

Let \(\hat{\mathbf{b}} = \mathbf{b}/\|\mathbf{b}\|\) be the unit vector along \(\mathbf{b}\). In the right triangle formed by \(\mathbf{a}\), its shadow and the residual, \(\mathbf{a}\) is the hypotenuse and the shadow is the side adjacent to angle \(\theta\). So the shadow’s (signed) length is

That number is the scalar projection (or component of \(\mathbf{a}\) along \(\mathbf{b}\)). It’s negative if \(\mathbf{a}\) points partly backwards. To get the projection as a vector, multiply this length by the unit direction:

The second form avoids square roots, which is handy by hand and in code. The perpendicular part is whatever’s left: \(\mathbf{a} - \operatorname{proj}_{\mathbf{b}}\mathbf{a}\).

Project \(\mathbf{a} = (2, 3)\) onto \(\mathbf{b} = (5, 1)\). Try it: Project a onto b.

In NumPy, where @ between two 1-D arrays is the dot product:

Take \(x\) horizontal and \(y\) vertical. The weight is \(\mathbf{W} = (0, -4 \times 9.8) = (0, -39.2)\) N. A unit vector pointing up the 30° ramp is \(\hat{\mathbf{u}} = (\cos 30^\circ, \sin 30^\circ) \approx (0.866, 0.5)\), and a unit vector perpendicular to the ramp, pointing out of its surface, is \(\hat{\mathbf{n}} = (-\sin 30^\circ, \cos 30^\circ) \approx (-0.5, 0.866)\). (Check: \(\hat{\mathbf{u}}\cdot\hat{\mathbf{n}} = -0.433 + 0.433 = 0\).)

These are the familiar Mechanics results \(mg\sin\theta\) and \(mg\cos\theta\), now obtained without drawing a single similar triangle. The ramp pushes back with a normal reaction of 33.9 N; friction must supply up to 19.6 N to stop the box sliding.

For resolving forces in a 2D Mechanics problem, the trigonometric shortcuts (\(mg\sin\theta\), \(mg\cos\theta\)) are fastest if you draw the diagram correctly. The dot-product method is mechanical and immune to "was it sine or cosine?" errors, works in 3D and beyond, and is what you’d write in code. Use trig for quick sketches; use projection for anything you’ll compute or anything not in a plane.

A search engine holds a million documents. A user types a query. Which documents are most relevant?

This is the founding problem of information retrieval, and since the 1960s the dominant answer has been geometric: turn every document and every query into a vector, and rank documents by how closely their vectors point the same way as the query’s.

The simplest representation is a bag of words. Fix a vocabulary of \(n\) words. A document becomes a vector in \(\mathbb{R}^n\) whose \(i\)-th component counts how often word \(i\) appears. With only two vocabulary words, "matrix" and "graph", we can draw it:

Should we rank by the raw dot product \(\mathbf{q}\cdot\mathbf{d}\)? It rewards documents that share words with the query, which is good. But \(\mathbf{q}\cdot\mathbf{d} = \|\mathbf{q}\|\,\|\mathbf{d}\|\cos\theta\) also grows with \(\|\mathbf{d}\|\), so a long document beats a short one just by being long. Paste a document into itself ten times and its score goes up tenfold, without becoming any more relevant.

The fix is to divide out the lengths, leaving only the angle. That’s cosine similarity:

It lies between \(-1\) and \(1\) (between 0 and 1 for count vectors, which have no negative entries), and depends only on direction.

Query \(\mathbf{q} = (1, 1)\) (the words "matrix graph"). Three documents: \(A = (10, 10)\), a long document using both words equally; \(B = (2, 0)\), a short one about matrices only; \(C = (1, 1)\), a short one using both. Try it: Dot product versus cosine.

The raw dot product says \(A\) is ten times as relevant as \(C\); cosine similarity correctly says they’re equally on-topic, and both beat \(B\), which covers only half the query.

Raw counts have another flaw: "the" appears everywhere and dominates every vector while telling you nothing. TF-IDF reweights each component by how informative the word is. A word’s document frequency \(\text{df}(w)\) is the number of documents containing it; out of \(N\) documents its inverse document frequency is \(\text{idf}(w) = \log(N / \text{df}(w))\), which is zero for a word in every document and large for a rare one. Each component becomes term frequency times IDF. Everything else, including cosine ranking, stays the same. Try it: Rank with TF-IDF.

The third document wins: it contains both query words, and "arrow" is rare (high IDF). The first scores a little for "vector". The second shares no query words and scores zero, its vector orthogonal to the query’s.

Bag-of-words vectors have a blind spot. "Car" and "automobile" are different vocabulary entries, so the corresponding axes are orthogonal, and a query for one scores zero against a document using only the other. The vectors capture spelling, not meaning.

Modern systems use embeddings: vectors produced by a trained neural network, designed so that texts with similar meanings get vectors pointing in similar directions. A typical sentence embedding has 384, 768, 1024 or more components. Here is the important point, and it’s worth slowing down for: mathematically, an embedding is exactly the same kind of object as \((3, 4)\). It’s a list of real numbers. Its length is \(\sqrt{\sum v_i^2}\), with 768 terms instead of 2. Its dot product with another embedding is \(\sum a_i b_i\), with 768 products instead of 2. The angle between two embeddings is \(\cos^{-1}\) of the normalised dot product, defined by the very formula we derived from the law of cosines. Nothing in our derivation used the fact that we were in two dimensions; we only used the algebraic rules of the dot product. You can’t draw a 768-dimensional arrow, but every calculation you’d do on the drawing still works.

What’s different is what the components mean. In \((3, 4)\) the components are "east" and "north". In an embedding, no single component has a clean meaning; meaning lives in directions, combinations of many components. Similar meanings point similar ways; that’s all the geometry needs.

You’ll meet the expression \(\mathbf{q}\cdot\mathbf{d}\) constantly in information-retrieval and machine-learning papers. You can now unpack it completely:

Here are five toy documents with 4-dimensional "embeddings" (real ones come from a model, but the search step is identical). Storing the documents as the rows of a 2-D array lets NumPy compute all five dot products in one operation.

Note the ranking: the pet documents cluster near the query; the finance ones point elsewhere. Dn @ qn is a matrix–vector product, which is just "dot the query with every row". We’ll see in Chapter 27 that this is one of the two ways to read matrix–vector multiplication.

The rule of thumb: use the similarity your embedding model was trained with. Its documentation will say.

Everything we’ve done extends algebraically to 768 dimensions. But our intuition was trained in two and three. Which mental pictures survive the trip, and which mislead?

Pick two random directions in the plane. The angle between them is equally likely to be anything, so their cosine similarity is spread widely between \(-1\) and \(1\). Now pick two random directions in 768 dimensions. Their cosine similarity is almost always within a few hundredths of zero: random high-dimensional vectors are nearly orthogonal.

Why? The dot product of two random unit vectors is a sum of \(n\) small terms, each as likely to be positive as negative. Like a random walk, the positives and negatives mostly cancel. The typical size of such a sum, relative to the maximum possible, shrinks like \(1/\sqrt{n}\) (a fact you’ll be able to prove after Chapter 30). You can see it directly: Try it: Random cosines in 2D.

This is good news and bad news. Good: in high dimensions there’s room for an enormous number of nearly independent directions, so a 768-dimensional space can encode many distinct concepts without them interfering. Bad: if your vectors are close to random, all similarities look alike, and "similar" becomes meaningless.

A 3D model is made of triangles. To light a triangle with Lambert’s law, you need its normal vector: a vector perpendicular to the triangle’s surface. You have two edge vectors \(\mathbf{a}\) and \(\mathbf{b}\) lying in the triangle. How do you find a vector perpendicular to both?

In three dimensions (and only there) there’s a product of two vectors that is a vector, perpendicular to both inputs:

A mnemonic for the pattern: for the first component, ignore the first entries and cross-multiply the second and third (\(a_2 b_3 - a_3 b_2\)); the other components cycle the indices \(1 \to 2 \to 3 \to 1\). (In Chapter 27 you’ll see it written as a determinant.)

Its geometric properties are what matter:

Compute \((1, 2, 3)\times(4, 5, 6)\) and check the result. Try it: Cross product (1,2,3)×(4,5,6).

The parallelogram has area \(\sqrt{9 + 36 + 9} = \sqrt{54} \approx 7.35\), so the triangle with those two edges has half that, about 3.67. NumPy agrees: np.cross([1, 2, 3], [4, 5, 6]) returns array([-3, 6, -3]).