Chris Collins

From transistor to CPU

2026-10-30T00:00:00Z

A computer is the most complicated thing most of us own, and it is made almost entirely of one very simple part, repeated. A 6502 — the processor in the Apple II, the Commodore 64, the BBC Micro and the Atari 800 — has about 3,500 transistors. A modern laptop chip has tens of billions. In both, each transistor does the same humble job: it is a switch that one wire can turn on and off.

This exploration climbs from that switch to a working processor in six steps, and at every step there is a live circuit on the page for you to poke. Nothing is hidden behind a diagram: each level is built, visibly, from the one below it. By the end you will have run a program on a CPU made of gates you can see, and then on a real 6502, and the two programs will look remarkably alike.

Step 1: a switch that a wire can flip

A transistor has three connections. In the bipolar kind used below, a small current into the base lets a much larger current flow from the collector to the emitter. Push a milliamp into the base and the transistor turns hard on; take it away and it turns off. For analogue work we care about the region in between, where the transistor amplifies. For digital work we deliberately avoid it: we only ever drive the transistor fully on or fully off.

Here is the simplest useful circuit there is. A pulse generator, Vin, swings between 0 V and 5 V every millisecond. It feeds the base of transistor Q1 through a 4.7 kΩ resistor. Q1’s collector pulls current through a red LED and a 220 Ω resistor from the 5 V supply.

What to try: scroll down inside the frame to the scope, then compare the two traces.Open full tool →

Scroll down inside the frame to the Transient (scope) plots. The orange trace is the input, the blue trace is vc, the voltage at the collector — the point between the LED and the transistor.

Try this:

Look at the two traces side by side. When vin is low, vc sits high, at about 4.2 V. When vin goes high, vc drops to almost nothing. The Currents plot shows why: about 13.5 mA flows through the LED only while the input is high.
Select Rb and change it to 100k, then press Run transient. Less base current means the transistor is no longer driven hard on, and vc no longer falls cleanly to zero. That in-between region is exactly what digital designers keep away from.
Put it back to 4k7 and change Rled to 1k. The LED current falls, but the shape of vc does not: on is still on, off is still off.

What to notice: the output is the opposite of the input. High in, low out; low in, high out. Read vc as the output and you have not just built a switch, you have built a logic gate — a NOT gate, or inverter. Everything else in this exploration is made of the same trick, applied over and over.

Real chips use MOSFETs rather than bipolar transistors, because a MOSFET’s gate draws almost no current, and they pair an n-type with a p-type transistor (CMOS) so that one is always off and the circuit draws almost no power when it is not switching. The palette has an NMOS part if you want to try building one. The idea is the same: a voltage on one wire decides whether another wire is connected.

Step 2: from switches to gates

Put two transistors in series, so current only flows when both are on, and the output is pulled low only when both inputs are high: that is a NAND gate. Put them in parallel, so either can pull the output low, and you have NOR. Add an inverter after either and you get AND and OR. From here on we stop drawing transistors and draw gates instead, the way chip designers do. It is the first abstraction on the climb, and it is a safe one: every gate below could be swapped for a handful of transistors without changing a thing.

What to try: flip A and B and watch all seven gates answer at once.Open full tool →

Two switches, A and B, feed seven gates. Bright wires and outputs are 1, dim ones are 0.

Try this:

With both switches off, notice that NAND, NOR, XNOR and NOT A are already lit. A gate whose name starts with N is the inverse of its namesake.
Flip A on. AND stays dark (it needs both), OR lights (it needs either), XOR lights (it wants them different).
Now turn B on too. XOR goes dark again, XNOR lights. XOR is the odd one out of the seven, and it is about to become important: it is the gate that adds.

What to notice: NAND on its own is enough to build every other gate — NOT is a NAND with its inputs tied together, AND is NAND then NOT, and so on. That is why early logic families sold chips full of nothing but NAND gates, and why "everything is NAND" is the usual first surprise in a digital course.

Step 3: gates that add

Add two one-bit numbers and there are only four cases: 0 + 0 = 0, 0 + 1 = 1, 1 + 0 = 1 and 1 + 1 = 10 in binary — that is, a sum bit of 0 and a carry of 1. Look at those sum bits: 0, 1, 1, 0. That is XOR. Look at the carries: 0, 0, 0, 1. That is AND. Two gates make a half adder.

It is only half an adder because it cannot accept a carry coming in from the column to its right. A full adder takes three inputs — A, B and a carry in — and makes a sum and a carry out. You can build one from two half adders and an OR gate.

What to try: read down the S and C columns, then the SUM and COUT columns.Open full tool →

This opens on the Truth table tab, which simulates the circuit for every combination of its inputs. The top part of the circuit is the half adder (inputs A, B, outputs S and C); the bottom part is a full adder packaged as a reusable component (inputs A2, B2, Cin, outputs Sum and Cout).

Try this:

Find the rows where A2, B2 and CIN are all 1. Both SUM and COUT are 1: one plus one plus one is three, which is 11 in binary.
Scroll down past the table: for each output the tool reads the logic back as an expression and minimises it. The full adder’s sum turns out to be a three-way XOR.
Switch to the Circuit tab and double-click the Full adder block to look inside it. A component is just a circuit with pins — the second abstraction on the climb.

Chain four full adders, each one’s carry out feeding the next one’s carry in, and you can add two four-bit numbers.

What to try: set A to 0111 and B to 0001 and watch the carry ripple up.Open full tool →

The switches come in pairs: A0/B0 is the lowest bit, A3/B3 the highest. The four sum bits and the final carry come out on the right, with the sum shown in hex.

Try this:

Turn on A0 and B1: 1 + 2. The display reads 3.
Now set A to 7 (A0, A1, A2 on) and B to 1 (B0 on). Every column produces a carry that the next column has to absorb, and the display jumps to 8.
Set every switch on: 15 + 15. The display shows E and Carry lights: 30 is 1 1110 in binary, one bit too many for four, so the fifth bit falls out as the carry.
Turn on Gate delays in the toolbar and repeat 7 + 1. Now each gate takes a moment, and you can watch the carry ripple from bit 0 to bit 3, the sum briefly showing wrong values on the way.

What to notice: this is a ripple-carry adder, and the ripple is its weakness. A 64-bit version has to wait for a carry to crawl through 64 stages. Real processors spend extra gates on carry-lookahead logic that works out all the carries at once. Speed in hardware is often bought with more transistors.

If you would rather write this than wire it, the Adders and hierarchy lesson of the HDL trainer has you build the same full adder and four-bit chain in Verilog, the language chips are actually designed in.

Step 4: memory from feedback

Everything so far has been combinational: the outputs depend only on the inputs right now. Turn the switches back and the answer goes back too. A computer also needs to remember — the number it is adding to, the instruction it is on — and the trick that gives gates a memory is wonderfully small. Feed a gate’s output back into its own input.

What to try: press Set, release it, and notice that Q stays on.Open full tool →

Two NOR gates, each feeding the other. Set and Reset are push buttons. When the page opens, the outputs are red: unknown. Nobody has told the circuit what to remember yet, and the simulator is honest about it.

Try this:

Hold Set for a moment and let go. Q goes to 1 and stays there, even though nothing is pressing it any more. Each gate is holding the other in place.
Press Reset. Q drops to 0, Q̄ rises, and that state holds too.
Press Set again while it is already set. Nothing changes. The latch only cares about which button was pressed last.

What to notice: that is one bit of memory, built from two NOR gates — eight transistors in CMOS. This SR latch is the ancestor of every register in a CPU. Add a little gating so that it only listens when a clock signal says so, and you have a flip-flop: a bit that changes only on the tick of a clock. Line up four flip-flops and you have a four-bit register. Wire their outputs back to their inputs through some logic and you have a circuit that steps through states on its own — the binary counter example does exactly that, counting 0 to 15 from four flip-flops and two AND gates.

The clock is the quiet hero of digital design. Between ticks, the combinational logic is free to glitch and ripple as much as it likes, as you saw in the adder. On the tick, every flip-flop captures the settled answer at once. A designer’s main job is to make sure the slowest path through the logic settles before the next tick; the length of that path sets the clock speed.

Step 5: a CPU you can see inside

We now have every ingredient. A register to hold a number. An adder to change it. A counter to step through a list. Some memory to hold that list. Put them together with one more idea, and you have a processor.

The idea is that the list is instructions. A counter, called the program counter or PC, points at a byte in a read-only memory. That byte is decoded: some of its bits say what to do (the opcode), the rest give a number to do it with (the operand). The decoded bits steer multiplexers — switches made of gates — that route the right values into the adder and the right result into the right register. On the next clock tick, the result is captured and the program counter moves on. Then it all happens again.

What to try: watch the two hex displays, then press Pause and step one clock cycle at a time.Open full tool →

This is a whole four-bit CPU in about two dozen parts. Find the pieces from earlier: the Counter on the left is the program counter, ROM holds the program, DEC decodes the opcode, the ALU is an adder that can also subtract, A is the accumulator register, C is the carry flag (a single flip-flop, with a yellow LED) and OUT is an output register. Hex displays show the PC, A and OUT. It starts running as soon as it loads.

The program in the ROM is four bytes long:

00   LDI 0      ; A ← 0
13   loop: ADD 3 ; A ← A + 3
20   OUT        ; OUT ← A
31   JMP loop   ; PC ← 1

Each byte is one instruction: the high hex digit is the opcode, the low digit the operand. So 13 means "opcode 1 (ADD), operand 3".

Try this:

Watch the OUT display, on the far right: 3, 6, 9, C, F, then 2. Four bits can only count to 15, so 15 + 3 wraps round to 2 — and the carry flag notes that it happened.
Press Pause, then Cycle to advance one clock period at a time. The display under the counter is the PC; watch it go 1, 2, 3, then jump back to 1.
Turn on Gate delays and run slowly. Every instruction becomes a little wave of activity spreading from the ROM through the decoder and ALU, settling before the next tick.
Select the ROM to edit the program. Try LDI 5, l: OUT, SUB 2, JC l, HLT. It counts 5, 3, 1 and then stops, because the third subtraction borrows, clears the carry flag, and the conditional jump JC falls through to the halt.

What to notice: the program is data. Nothing in the hardware changed when you edited the ROM, yet the machine did something different. That is the whole point of a processor, and it is why the same chip can run a word processor, a game, or this web page. The second program also shows the other essential ingredient: a conditional jump. A machine that can choose what to do next based on a result is, given enough memory, able to compute anything computable.

Step 6: a real 6502

The 6502 was launched in 1975 at a price of $25, when its rivals cost several times as much, and it ended up in a huge share of the home computers and consoles of the next decade. It is an eight-bit processor rather than a four-bit one, with three registers (A, X and Y), a stack, more addressing modes, and around 150 opcodes instead of eight. But at heart it is our tiny CPU, grown up: a program counter, a decoder, an ALU, an accumulator and a handful of flags.

Here is the "count in threes" program again, written for the 6502.

What to try: read the listing, the address and the bytes of each instruction, then compare it with the tiny CPU's ROM.Open full tool →

The listing on the right shows each instruction’s address and the bytes it assembles to: A9 00 for lda #0, 18 for clc, 69 03 for adc #3, 8D 00 02 for sta $0200 and 4C 02 06 for jmp loop. Eleven bytes in all.

Try this:

Compare it with the tiny CPU’s ROM. Load, add, output, jump: the same four ideas. The 6502’s instructions are longer because its addresses are sixteen bits, so jmp loop needs two bytes to say where loop is — 02 06, low byte first.
Notice the extra clc (clear carry). The 6502 has no plain add: adc always adds the carry flag too, so that a sixteen-bit sum can be built from two eight-bit ones. Delete the clc and the program may count in fours whenever the previous sum overflowed.
Switch to the Opcode table tab and filter for ADC. There are eight versions of it, one for each way of finding the number to add, each with its own cycle count.

Assembling is only half of it. Below is an Atari 800XL, emulated in WebAssembly, with a 6502 that runs the real Atari operating system. The example program is fifteen lines long and does something no amount of reading quite prepares you for.

What to try: press Assemble & run, then read the loop on lines 14 to 19.Open full tool →

Try this:

Press Assemble & run. The machine powers on, the program is assembled and loaded at $2000, and the screen fills with scrolling bands of colour.
Read the loop. It reads VCOUNT, the number of the scan line the TV beam is drawing; adds the frame counter so the pattern moves; then sta WSYNC, which freezes the CPU until the beam reaches the end of the line; then stores the value in COLBK, the background colour. Every coloured stripe on the screen is one trip round a six-instruction loop.
Press Pause and open the Debugger tab. The registers you see there — PC, A, X, Y, S and the flags — are the 6502’s equivalents of the tiny CPU’s counter, accumulator and carry flip-flop. Press Step and watch the PC move through the loop one instruction at a time.
Replace adc RTCLOK with adc #0, or add asl a after lda VCOUNT to double the line number, and run it again. Small edits, very different pictures.

What to notice: that loop is racing a television. Each 6502 instruction takes two to six clock cycles, at about 1.8 MHz, and a scan line lasts 114 cycles. Programming these machines meant counting cycles the way we counted gate delays in the adder: the logic has to settle before the next tick, only now the tick is the edge of the picture.

The climb, looking back

Six steps, each a small idea:

A transistor is a switch controlled by a wire, and wired the right way it is a NOT gate.
A few transistors make NAND and NOR, and NAND alone can make every other gate.
XOR and AND make a half adder; two of those make a full adder; a chain of them adds numbers.
Feedback turns gates into memory; a clock makes that memory change in step.
A counter, a ROM, a decoder, an adder and a register make a processor, and the program is just data in the ROM.
Grow every part — wider words, more registers, more opcodes — and you have a 6502. Grow it again, with caches, pipelines and billions of transistors, and you have the chip you are reading this on.

At no point did anything magic happen. Each level is the one below it, wrapped up and given a name. The only genuinely surprising thing is how few kinds of idea it takes.

Where to go next

Digital logic simulator — build your own circuits, or open the traffic-light state machine, seven-segment decoder and glitch examples.
Boolean logic — minimise expressions and draw the Karnaugh maps behind the adder’s equations.
Circuit simulator — the transistor level, with real device models and ngspice to compare against.
HDL trainer — learn Verilog from wires and gates up to UARTs, VGA and FIFOs.
FPGA playground — a small CPU in Verilog, synthesised and placed on a real FPGA in the browser; the RISC-V example goes further.
6502 assembler — assemble, disassemble and browse every opcode with its cycle counts.
Atari 8-bit computer — more 6502 examples, from display-list interrupts to player-missile graphics, with a full debugger. The 6502 also lives in the Apple II, Commodore 64 and BBC Micro tools.

How a brushless motor spins

2026-10-30T00:00:00Z

Pick up the motor from a racing drone and turn it with your fingers. It is not smooth: it clicks gently from one resting place to the next, as if it had little detents. Spin it hard and it hums. There are no brushes, no commutator, nothing touching anything except the bearings. Three wires come out of it, and on its own it will do absolutely nothing — connect it straight to a battery and you get a twitch, a lot of current and a warm motor.

Everything interesting about a brushless motor lives in the gap between that inert lump of copper and magnets and the controller that drives it. This exploration walks through both, from the windings to the switching pattern to the trick that lets the controller find the rotor without any sensors, and ends with four of these motors holding a quadcopter in the air. The simulators are embedded as we go. They all run in your browser, and nothing you do in them can burn anything out, which is more than I can say for my first ESC.

Magnets go round, coils stay still

An ordinary brushed DC motor has its coils on the spinning part and its magnets in the case. To get current into coils that are turning, it uses brushes rubbing on a segmented copper ring, the commutator. As the rotor turns, the brushes slide from segment to segment and the current is switched into whichever coils are in the right place to keep pulling. It is clever, mechanical and self-timing, and it is also the thing that wears out, sparks and limits speed.

A brushless motor turns this inside out. The coils sit still, wound around the teeth of a laminated iron stator, and the magnets are on the rotor. In a drone motor the rotor is the bell on the outside — an outrunner — with a ring of curved magnets glued inside it. Nothing that carries current moves, so there is nothing to wear. The price is that the commutator has gone: something else now has to switch the current between coils at exactly the right moments. That something is the electronic speed controller, the ESC.

Before we get to the switching, look at the copper. Here is a typical 5-inch quad motor, a "2207" (a 22 mm stator, 7 mm tall) wound for 1,950 Kv:

What to try: watch which coils light as the bell turns; then change Poles from 14 to 8 and see the winding pattern, winding factor and cogging steps change.Open full tool →

The drawing shows twelve teeth (slots) and fourteen magnets (poles): this is 12N14P, the combination on almost every multirotor motor. Each tooth carries a coil, coloured by the phase it belongs to — red, amber and blue for A, B and C — and the pattern written above the picture, AabBCcaABbcC, reads round the stator from the top. A capital letter is a coil wound one way round its tooth, a small one is wound the other way.

Try this:

The rotor is already turning, with the coils lit as an ESC would switch them (Pause stops it). Watch how far the bell moves during one full cycle of the lighting pattern: one pole pair, two magnets' worth, about 51°. Fourteen magnets is seven pole pairs, so the electrical cycle runs seven times for every turn of the shaft. That is why the panel says 25,000 rpm is 175,000 eRPM — electrical revolutions per minute — a number you will meet again in a moment.
Read the cogging steps per turn: 84, the lowest common multiple of 12 and 14. Those are the little detents you felt with your fingers. Many small steps make for smooth running; a combination like 12N8P has only 24, and they are much stronger.
Change Poles to 8, then 10, then back to 14. The winding factor — how much of each coil’s effort actually adds up in the phase, the designer’s efficiency of copper — is 0.866 for 12N8P and 0.933 for both 12N10P and 12N14P. The cogging steps go from 24 to 60 to 84: same copper, smoother motor.
Switch Layout to "alternate teeth". Half the teeth go bare. This is the LRK winding: easier to wind by hand, slightly different character.

The important idea to carry forward is that the stator can make a magnetic field that points in any of several directions, depending on which phases carry current and which way. If you can make that field point a little ahead of the rotor’s magnets, the magnets chase it. Keep moving the field and the rotor keeps chasing. That is the whole motor.

Six steps round the circle

Three phases, each of which can be pulled up to the battery, pulled down to ground, or left floating. The ESC does this with six MOSFETs, a high and a low switch per phase — a three-phase bridge. The simplest way to drive it is to always have exactly one phase high, one phase low and one floating. There are six ways to choose that, and stepping through them in order rotates the field round the stator in six 60° jumps. This is six-step or trapezoidal commutation, and it is what nearly every hobby ESC did for twenty years.

The simulator reduces the motor to the simplest possible picture — two poles, three coils — so that one electrical cycle is one turn of the drawing:

What to try: follow the highlighted row of the table as the rotor turns, then scroll to the terminal-voltage plot and find the dotted zero crossings.Open full tool →

The magnet is the red and grey bar. The arrow is the stator field. The three dots around the edge are Hall-effect sensors, each filling in when it sees a north pole, and the table on the right lists the six steps: which Hall code is showing, which phase is switched high, which low, and which is left floating.

Try this:

Pause the animation and step through it by eye. The field arrow always sits somewhere between 60° and 120° ahead of the magnet’s north pole — 90° on average — and jumps 60° at every commutation. Torque is greatest when the field is at right angles to the magnet, so six-step spends its whole life a little either side of ideal.
Look at the three Hall bits. Of the eight codes three bits can make, only six ever appear: 000 and 111 mean a sensor has failed. Three cheap sensors, 120° apart, are enough to know which sixth of a cycle the rotor is in, and that is all six-step needs.
Scroll down to One electrical cycle. The phase currents are blocks — positive for 120°, off, negative for 120°, off — with their corners rounded by the coil’s inductance. Now look at the torque trace: it dips at every commutation. The summary above it puts a number on it, about 30% torque ripple for this motor. You hear that ripple as part of the motor’s whine.
Change Back-EMF shape from sinusoidal to trapezoidal. The ripple drops: block currents suit a motor whose back-EMF is flat-topped. Most drone motors are closer to sinusoidal, which is a hint of where we are going.

Hearing the rotor without sensors

Drone motors have no Hall sensors. They spin at tens of thousands of rpm in a cloud of dust, next to a lot of switching noise, and every gram matters. So how does the ESC know when to commutate?

The motor tells it. A magnet sweeping past a coil induces a voltage in it — the motor is also a generator — and this back-EMF rises with speed. Kv is just that relationship turned round: a 1,950 Kv motor generates one volt for every 1,950 rpm, and on a 22.2 V battery it would spin up to a little over 43,000 rpm with no load, the speed at which its own back-EMF cancels the supply. In six-step there is always one phase floating, not driven by the ESC, and the voltage on a floating phase is its back-EMF (plus the star point it sits on). When that back-EMF crosses the midpoint — a zero crossing — the rotor is exactly halfway through the step. The ESC waits another 30° electrical, judged from the time between the last two crossings, and switches.

Go back to the six-step view above and scroll to the terminal voltage plot. The sloping lines are the floating phases' back-EMF, rising or falling across the gap between the flat driven levels. The dotted verticals mark where each crosses the star-point voltage; the dashed ones, 30° later, are the commutations the ESC schedules from them.

Try this:

Notice the short vertical spike just after each commutation, where the phase that was switched off is clamped to 0 V or the full bus by a diode until its current dies away. A real ESC’s comparator has to ignore that window, or it sees a false zero crossing and stumbles. Desyncs — the motor suddenly stuttering in a hard punch — are often exactly this.
Type a ZC filter delay of 10 µs. The commutations slide late, the field falls behind the ideal angle, and torque and efficiency drop. At 175,000 eRPM one electrical cycle lasts only about 340 µs, so 10 µs is already several degrees.
Now set a Timing advance of 15°. The ESC commutates early to make up for delays and for the inductance that slows the current’s rise. The Timing advance chart at the bottom sweeps it for you: at this speed, more advance buys torque. BLHeli and Bluejay’s "timing" setting is this number.
Lower the Speed to 2,000 rpm. The back-EMF shrinks to a few volts and the crossings get hard to see — which is why a sensorless motor has to be started open-loop, kicked round blind until it is fast enough to hear itself. It is also why sensorless motors are rough at very low speed, and why an RC car, which has to crawl away from a standstill under load, often has sensors.

Field-oriented control: steering the field smoothly

Six-step is cheap and effective, but it chops a smooth rotating field into six lurches. The better answer is to make the stator field rotate continuously, always exactly at right angles to the rotor’s magnets, with just enough current to make the torque you want. That is field-oriented control, FOC, and it is what VESC, ODrive, most e-bike controllers and a growing number of drone ESCs (AM32 among them) do.

The difficulty is that three sinusoidal phase currents are awkward things to control — they never sit still. FOC’s trick is two changes of point of view. The Clarke transform adds the three currents (which always sum to zero) into a single vector on two fixed axes, α and β. The Park transform then looks at that vector from the rotor itself, on axes d (along the magnet) and q (at right angles to it). In the rotor’s frame, a perfectly driven motor’s currents stop rotating and become two constant numbers.

What to try: watch the three phase currents swirl while id and iq stay flat; then scroll to the six-step, sine and FOC comparison.Open full tool →

Try this:

Watch the vector diagram. The three coloured arrows — each phase’s contribution — change length all the time, but they always add up to the same black vector, which turns with the rotor. In the table, ia, ib and ic change continuously; id stays at 0 and iq stays at 10 A. Those two steady numbers are what the controller actually regulates.
Set the id reference to −5. The current vector leans back towards the magnet’s south pole and starts to weaken the magnet’s field. That is field weakening: it costs current but lets the motor run above the speed its back-EMF would otherwise allow.
Scroll to Six-step, sine and FOC at the same speed and torque. All three are asked for the same torque at 25,000 rpm. FOC makes it with the least current and no ripple at all; six-step gets there with about 30% ripple and slightly more current; and plain sinusoidal voltage drive, with no current feedback, cannot make the torque at all at this speed, because the coil’s inductance makes the current lag the voltage and nobody corrects for it. That one row is the whole case for measuring the currents.
At the bottom, The d and q current loops are two ordinary PI controllers, one for each axis. Untick Decoupling feed-forward and step again: the d current now kicks when q moves, because at speed the two axes leak into each other. Then try a loop bandwidth of 200 Hz and of 5,000 Hz and see how quickly iq reaches its target.

FOC is more arithmetic — a sine, a cosine and two multiplications per axis, twenty thousand times a second — and it needs current sensors. A modern 32-bit microcontroller shrugs at both, which is why it is slowly taking over.

How the flight controller speaks to the ESC

A multirotor’s flight controller decides, thousands of times a second, how hard each motor should push. It used to say so with a servo pulse — 1,000 to 2,000 µs wide, fifty times a second — which the ESC had to measure and calibrate. Then came OneShot and MultiShot, the same idea with shorter pulses. Now almost everything uses DShot: the number itself, sent as bits.

What to try: drag the throttle slider and watch the DShot frame change bit by bit; then scroll down to the eRPM telemetry.Open full tool →

A DShot frame is sixteen bits: an 11-bit throttle value (48 to 2,047; the values below 48 are commands such as "beep" or "reverse direction"), one bit asking for telemetry, and a 4-bit checksum. At DShot600 each bit takes 1.67 µs, so a whole frame is under 27 µs — no calibration, no jitter, and a corrupt frame is simply ignored.

Try this:

Move Throttle (%) and watch the blue value bits and the green checksum. Switch Send to a command and send "1: Beep 1" — this is how your quad beeps when you have lost it in long grass.
Untick Bidirectional. The checksum flips. With bidirectional DShot the line is inverted and, after each frame, the ESC talks back on the same wire.
Scroll to eRPM telemetry. What the ESC sends back is the electrical period — from its zero-crossing timing, the same thing it measures to commutate — and the flight controller divides by the pole pairs to get shaft speed. 140,000 eRPM on a 14-pole motor is about 20,000 rpm. Set Motor poles to 12 and the same reply reads as a different speed: this is why Betaflight’s motor_poles must match your motor, or its RPM filter notches the wrong frequencies.

Four motors and a battery

A motor alone is only half an answer. What it does depends on the propeller it turns and the battery that feeds it, and the three pull against each other: a bigger prop makes more thrust and draws more current; the current makes the battery sag; the sag slows the motor. The RC power solver finds the point where they agree.

What to try: drag the battery from 4 to 6 cells and watch current, thrust and the warnings; then change the propeller pitch.Open full tool →

It opens on a 5-inch freestyle quad: four 2306 2,450 Kv motors, 5.1 × 4.3 × 3 props, a 4S 1,500 mAh pack, 650 g all up. Four kilograms of static thrust for a 650 g machine, a thrust-to-weight of about 6, hovering at a little over a quarter throttle. Then the warnings: at full throttle each motor is just over its current rating, and the pack is asked for more than 170 A, beyond what its C rating promises, sagging to around 3 V a cell.

Try this:

Change Cells in series to 6 without touching anything else. The motors spin half as fast again and the current goes through the roof. This is why 6S quads use lower-Kv motors — the 1,950 Kv motor we started with is a 6S motor, and 2,450 Kv is a 4S one. Set Kv to 1,700 and see things calm down.
Back on 4S, raise the prop pitch from 4.3 to 5. More thrust at speed, more current everywhere. Look at g per W in the hover column: that is your flight time.
Switch the sweep chart to Efficiency. A motor is most efficient well below full throttle, which is happy news for a quad that spends most of its life hovering at 30%.

Putting it in a frame

The last step is the whole aircraft. A quad is the motors, props, frame, flight controller, ESCs, camera, video transmitter and battery, and every gram of those has to be lifted.

What to try: add the action camera (set its quantity to 1) and watch the all-up weight, thrust-to-weight and flight time below.Open full tool →

The template is a 5-inch freestyle build on 6S. Scroll below the parts list to Frame, power and battery, where the full-throttle thrust per motor goes in — from a maker’s thrust table or from the solver above — and the tool works out all-up weight, thrust-to-weight, hover throttle and flight time, and checks the props clear each other on the frame.

Try this:

Set the Action camera quantity to 1. Seventy-five grams is a lot on a quad whose parts add up to 535 g: watch the hover throttle rise and the flight time fall.
Swap the 1,300 mAh battery for a heavier 1,800 mAh one (about 270 g). Flight time goes up, but by less than the capacity would suggest, because you are now carrying the extra cells too.
Make the Wheelbase 200 mm with 5-inch props and look at the prop clearance: the discs nearly touch.

One more loop

There is a control loop at the heart of FOC, and another wrapped around the whole quad: the flight controller compares the rotation rate the gyro measures with the rate your sticks ask for, and adjusts the four motors' throttle to close the gap. Both are PID controllers, and the same few ideas govern both.

What to try: raise Kp until the response rings, then set the dead time to 0 and see the same gain settle cleanly.Open full tool →

The plant here is generic — a gain, a time constant and a delay — but the behaviour is the one you tune on a quad.

Try this:

Raise Kp from 1.5 to 3. The response overshoots and rings. On a quad, that is the oscillation you hear and see in the footage when the P term is too high.
Set Dead time to 0. With no delay, the same high gain settles cleanly. Delay is what makes control hard — which is exactly why ESC latency, gyro filtering and the speed of DShot all matter to how a quad flies.
Set Ti blank (no integral) and look at the load disturbance at 50 s: the output never quite comes back. On a quad, I is what holds the attitude against a steady gust or an off-centre battery.

What to take away

A brushless motor is a ring of magnets chasing a rotating magnetic field. The windings decide how smooth and strong that chase is; the ESC decides where the field points. Six-step jumps the field round in 60° steps, timed by Hall sensors or by listening for zero crossings in the back-EMF of the floating phase. FOC measures the currents and rotates the field smoothly, exactly ahead of the magnets. And once you have the motor, the propeller and the battery decide what it can actually do, and a PID loop decides how well it is used.

Where to go next

Brushless motors — the full tool, with the Motor tab’s torque–speed and efficiency maps, Measure it for finding your own motor’s Kv and resistance with a drill and a meter, and the ESC tab’s MOSFET losses and bus capacitor ripple.
RC power setup — compare three motor, prop and battery setups side by side, for planes as well as multirotors.
FPV quad — the build planner, Betaflight rates, a CLI diff and a blackbox log viewer that plots the motor traces this page explained.
PID loop simulator — tuning rules and the controller as C code.
H-bridge — the brushed motor and single-phase bridge that the three-phase inverter grows out of; LiPo — the batteries on the other end of the wires; motion and drive — gearing a motor for a robot joint.

How a database survives a crash

2026-10-30T00:00:00Z

Here is a promise every serious database makes, and almost nobody stops to think about: once it says COMMIT succeeded, your data is safe. Not "safe unless the power goes", not "safe unless the kernel panics in the next forty milliseconds" — safe. And the transaction that was halfway through when the lights went out? Gone, cleanly, as if it had never started.

That is a strange thing to be able to promise. Disks write in blocks of 4 KB or so; a table and its indexes are spread over thousands of those blocks; a single UPDATE might touch a dozen of them. The power can fail between any two. So how does the database come back up with every block consistent?

The answer is one idea, applied relentlessly — write down what you are going to do before you do it — plus a lot of engineering to make that idea cheap. This exploration walks through it with the site’s database tools running live in the page. Nothing connects to a server: the PostgreSQL and SQLite you will poke at are real engines compiled to WebAssembly, and the storage and isolation simulators are models built to show the mechanisms.

The problem: pages in memory, pages on disk

A database keeps its data in fixed-size pages — 8 KB in PostgreSQL, 16 KB in InnoDB, 4 KB by default in SQLite. To change a row it reads the page into memory, changes it there, and at some point writes the page back.

Reading pages is slow, so databases keep a cache of them: the buffer pool. A page that has been changed in memory but not yet written back is dirty. The buffer pool would love to keep dirty pages around for as long as possible — if the same page is updated a hundred times, it only needs writing once — but every dirty page is a change the disk does not know about yet.

So we have a tension. For speed, we want to delay writing pages. For safety, we want every committed change on disk before we say "committed". Writing every touched page at every commit would be correct and ruinously slow: random writes all over the disk, and a torn mess if the power fails halfway through a page.

The write-ahead log

The way out is the write-ahead log (WAL). Before a page is allowed to change on disk, a record describing the change is appended to a log — a single file that only ever grows at the end. At commit, the database makes sure the log is on disk up to and including that transaction’s commit record (an fsync), and then it says yes. The pages themselves can be written whenever it is convenient.

Why is that better? Because appending to one file is sequential I/O, which even spinning disks do quickly, and because a log record is small: "on page 4117, slot 3, set column 2 to 150" is tens of bytes, not 8 KB. And if the machine dies, the log is the truth: replay it from the last known-good point and every committed change is redone; a transaction with no commit record is undone or simply ignored.

You can watch PostgreSQL’s log grow. The console below is real PostgreSQL 18, running in your browser through PGlite. pg_current_wal_lsn() returns the log sequence number — the byte position at the end of the WAL — so the difference between two of them is how many bytes of log a piece of work produced.

What to try: scroll through the results and compare the three numbers, then change the 10000 and press Run.Open full tool →

The script measures three things. First, inserting 10,000 rows produces about 1.6 MB of log — more than the table and its primary-key index occupy. That is the price of writing everything twice: once to the log now, and once more to the table’s pages later.

The second and third numbers are the interesting ones. After a CHECKPOINT, the first change to a page writes far more log than the second: when I ran it, about 19,000 bytes against about 300. Both updates change one short text value. Why the difference?

Try this:

Press Reset (so the tables are fresh), delete the CHECKPOINT; line and press Run. The first update no longer pays the big price, because no checkpoint has happened since its page was last logged.
Reset again, put the checkpoint back and change WHERE id = 2 to WHERE id = 9000: a row on a different page. Now the "second" update is also a first change to its page, and it costs about 16,000 bytes too.

What you are seeing is full-page writes, and it is about torn pages.

Torn pages and checkpoints

A PostgreSQL page is 8 KB; the disk underneath may only guarantee that 512-byte or 4 KB sectors are written atomically. If the power fails while a page is being written back, the page on disk can be half old and half new — torn. A log record that says "set slot 3 to 150" is no use against a torn page: it describes a change to a page that no longer exists in any consistent form.

PostgreSQL’s answer is to put a complete copy of the page into the WAL the first time the page changes after each checkpoint. Recovery starts from that checkpoint, so for every page it touches it first finds a whole, known-good image in the log, restores it, and then replays the small records on top. The image costs about one page of log (your ~19,000 bytes were two or three page images: the table page, the index page and the mark table’s page); every later change to the same page, until the next checkpoint, is just the small record.

A checkpoint, then, is the database saying: "every dirty page that existed when I started has now been written to disk, so recovery never needs to look further back than here." It bounds how much log has to be kept and how long recovery takes, and it resets the full-page-write clock. Checkpoints too often and you write page images constantly; too rarely and recovery after a crash takes a long time. That is a real tuning knob (checkpoint_timeout, max_wal_size).

The storage-engines simulator draws all of this: the WAL, the buffer pool with its dirty pages, and the disk, with a counter of every byte written. This view runs its update-heavy scenario on the B+tree alone with full-page writes switched on.

What to try: press Start, then Next step twice. Watch the WAL and disk boxes and the write amplification counter.Open full tool →

Try this:

After step 2 (Skewed updates) look at write amp. — with no buffer pool, every update of a 116-byte entry writes a whole page, and full-page writes add page images to the log on top. It lands in the high thirties.
Step 3 turns on a 128-page buffer pool. Hot pages now absorb many updates in memory and are written once, at eviction or checkpoint. Watch the write-amplification chart bend downwards.
Tap the WAL box or the buffer pool to inspect them; Checkpoint is one of the actions. Dots on pages show which are cached, orange ones are dirty.

This is the whole bargain of a classical database engine: the buffer pool makes writes cheap by delaying them, the WAL makes the delay safe, and checkpoints stop the WAL growing for ever.

Two ways to organise the pages: B+tree and LSM-tree

So far the pages have been the leaves and branches of a B+tree: one big sorted tree, updated in place. PostgreSQL’s indexes, InnoDB’s tables and SQLite’s files all work this way. Updating in place is what makes the WAL and the torn-page dance necessary.

There is another family that takes the log idea to its logical conclusion. A log-structured merge-tree (LSM-tree) — RocksDB, LevelDB, Cassandra, ScyllaDB, the storage under many modern systems — never updates a file on disk at all. A write goes to the WAL (for durability, exactly as before) and into a sorted structure in memory, the memtable. When the memtable is full it is frozen and written out in one sequential pass as an immutable sorted file, an SSTable. Its part of the WAL can then be thrown away.

What to try: press Start and step through all four steps; tap the memtable and an SSTable to inspect them.Open full tool →

Try this:

In step 1, notice that a write touches only two things: the WAL and the memtable. Nothing on disk is modified in place, so there is no page to tear.
In step 2 the memtable fills (six entries here — millions in real life), becomes the immutable memtable, and is flushed to level 0. The WAL’s live size drops: those records are no longer needed for recovery, because the data is now in a file.
In step 4, follow a lookup: memtable first, then the L0 files newest to oldest. A ✕ means a Bloom filter said "definitely not here" and the file was skipped without a disk read.

Crash recovery in an LSM-tree is pleasingly simple. The SSTables are immutable and were fsynced before they were recorded in the manifest, so they are either fully there or not there at all. The only thing lost in a crash is the memtable — and the WAL holds exactly the writes that were in it. Replay the WAL into a new memtable and you are back.

Compaction: paying the bill later

An LSM-tree buys fast writes by deferring work. Every flush adds another file, files overlap, and an updated or deleted key may now have several versions scattered across them. Reads get slower the more files there are, and old versions waste space. So, in the background, the engine compacts: it merge-sorts files together, keeps the newest version of each key, and writes fewer, bigger, non-overlapping files further down.

What to try: press Start, then Next step. Watch L0 fill to four files and get merged into L1, and L1 spill into L2.Open full tool →

Try this:

Watch the level labels: 30/32 means L1 holds 30 entries against a target of 32. When a level goes over its target, one of its files is merged with everything it overlaps in the level below.
Notice L0’s files overlap each other (they span most of the key range), while L1 and L2 are tidy rows of non-overlapping files. That is the payoff: below L0, a lookup reads at most one file per level.
In step 3, look at the LSM’s write amplification. Each entry is rewritten once per level it passes through — the cost of keeping reads cheap.

Compaction also matters for crashes, in a quiet way: it, too, writes new files and only then records them in the manifest and deletes the old ones. A crash mid-compaction leaves some orphaned half-written output, which is simply ignored and deleted on restart. Immutable files make "atomic" cheap.

If you want to see the two families side by side, the simulator’s Range scans favour the B+tree and Bloom filters scenarios are good next stops; the trade-off is usually summarised as read, write, space — pick two.

SQLite: atomic commit in a single file

SQLite is a useful contrast, because it has two complete answers to the crash problem and lets you switch between them with one PRAGMA.

In the default rollback journal mode, SQLite does the opposite of a WAL: before changing a page in the database file, it copies the page’s original content into a separate -journal file and fsyncs that. Then it writes the new pages into the database. If the process dies, the next program to open the database finds a "hot" journal and copies the old pages back. The database is rolled back to the last commit. The commit itself is the moment the journal is deleted.

In WAL mode, the database file is left alone during a transaction; new versions of pages are appended to a -wal file, readers look there first, and a checkpoint later copies them home. That is the same shape as PostgreSQL’s WAL, with whole pages as the records.

What to try: press Run every step, then read the header table under each step — offsets 18, 19 and 24 are the ones that change.Open full tool →

Try this:

Under step 1, find the file change counter at offset 24. Every write transaction in rollback mode bumps it; other processes use it to notice their cached pages are stale.
Under Switch to WAL, bytes 18 and 19 change from 1 (legacy) to 2 (WAL). The journal mode is stored in the file itself, so it persists.
Open the Pages tab afterwards and look at the map of the file: every page is accounted for, and the header at the start of page 1 describes the whole database.

The lab is honest about a limit: one browser page cannot pull the plug on itself, so what you see is the on-disk format the recovery logic relies on rather than a crash. But the mechanism is all here — a page either has a known-good copy somewhere or it was never touched.

The other half of the promise: isolation

Surviving a crash is the D in ACID — durability — and its sibling A, atomicity. But a database that is merely durable can still hand your application nonsense, if two transactions run at the same time and see each other’s half-finished work. That is the I: isolation.

The machinery is related. Modern databases keep several versions of a row (MVCC — multi-version concurrency control), and each transaction reads from a snapshot: the set of versions committed when it started (or, at READ COMMITTED, when each statement started). Old versions are what the undo log or the heap’s dead tuples are for. The isolation level decides which snapshot you get and what happens when two transactions want the same row.

The isolation lab steps two sessions through an interleaving, statement by statement, against models of PostgreSQL and InnoDB that have been checked against the real servers.

What to try: press Step eight times, then change the isolation level to REPEATABLE READ and run it again.Open full tool →

Try this:

At READ COMMITTED, both sessions read a balance of 100, A writes 150, B writes 125. B waits for A’s row lock, then overwrites. The banner says The anomaly happened: one deposit is lost.
Switch to REPEATABLE READ. B’s UPDATE now fails with 40001 could not serialize access due to concurrent update. The application has to retry from BEGIN — and on the retry it reads 150.
Switch the engine to MySQL InnoDB at REPEATABLE READ. The lost update happens. Same SQL standard name, different behaviour; this is why the lab exists.

The lost update has well-known fixes — SELECT … FOR UPDATE, an atomic SET balance = balance + 25, a version column — and they are all in the scenario list. The nastier anomaly is the one no row lock can see.

What to try: step to the end at REPEATABLE READ, then switch to SERIALIZABLE and step through again.Open full tool →

Try this:

Alice and Bob are both on call for shift 1. Each checks that two doctors are on call, and each takes themselves off. They update different rows, so nothing conflicts and the snapshots are perfectly consistent. At the end, nobody is on call — the Data panel shows both false.
At SERIALIZABLE, PostgreSQL’s Serializable Snapshot Isolation tracks what each transaction read as well as what it wrote. It spots the dangerous pattern and fails B’s commit with 40001. Read the explanation under Step 8.
Try MySQL InnoDB at SERIALIZABLE: there, plain reads take shared locks, and the two updates deadlock instead. Prevented, but by a different route.

Finally, the whole map at once. Every scenario, both engines, all four levels:

What to try: find the Write skew row and read across it; click any cell to step through that combination.Open full tool →

Try this:

Red cells are anomalies that happen; dark green means the database stopped it with an error you must retry; pale green means it was stopped by waiting or by the snapshot.
Compare the PostgreSQL RR column with the MySQL RR column. They share a name and very little else.
Notice how much green at SERIALIZABLE is an error. Serialisable isolation is not free safety: it is safety in exchange for retry loops in your application.

Putting it together

Follow one UPDATE accounts SET balance = 175 WHERE id = 1; COMMIT; from start to finish:

The transaction takes a snapshot and a row lock; the isolation level decides what it may see and whether a concurrent writer makes it wait or fail.
The page holding the row is found in the buffer pool (or read into it) and changed in memory. It is now dirty.
A WAL record describing the change is appended to the log buffer — with a full-page image if this is the page’s first change since the last checkpoint.
At COMMIT, the log is fsynced up to the commit record. Only then does the client hear "OK".
Some time later — at eviction, or at the next checkpoint — the dirty page is written to its home on disk.
If the power fails anywhere in that sequence, recovery starts at the last checkpoint, restores page images, replays every logged change, and discards the work of transactions that never committed.

An LSM-tree swaps steps 2 and 5 for a memtable and a flush, then spends its spare time compacting; SQLite in rollback mode inverts the log and saves the old pages instead of the new ones. But every one of them rests on the same rule: before you change the only copy of something, write down enough to get back.

Where to go next

Storage engines — all ten scenarios, including size-tiered compaction’s space spike, tombstones, and the Cost models tab with the textbook formulas.
Postgres playground — real PostgreSQL in the browser, with pattern labs for queues, advisory locks, leases and an animated concurrency model.
SQLite internals — the file decoded byte by byte, b-tree diagrams, bytecode, and ten more labs.
Isolation lab — every anomaly, the fixes, the Hermitage tests, and a mode that runs the same steps on your own PostgreSQL server.
Consistency lab — the same questions once the database is replicated: stale reads, session guarantees and linearisability.
Transactions across services — what you can do when one database’s WAL is no longer enough: two-phase commit, sagas and the outbox.
AOEE, a TAO-style social graph cache in Rust — a cache in front of a database, and what that does to consistency.

How a jet engine makes thrust

2026-10-30T00:00:00Z

A jet engine looks like magic from the departure lounge: a tube that swallows air and pushes a hundred tonnes of aeroplane to 900 km/h. Close up it is one of the most satisfying machines to understand, because every part of it does one job you can name, and the whole thing reduces to a single idea — throw air backwards faster than it arrived.

This exploration follows a parcel of air through an engine, from the intake to the nozzle. At each stop there is a live view of the jet engine tool working the numbers out as you watch, and some things to try. The numbers are from a real cycle analysis, the same method the propulsion textbooks use, so when you change something the engine answers honestly. Near the end we shrink the whole thing down and bolt it to a car.

Thrust is momentum

Newton’s third law does all the work. If an engine takes in ṁ kilograms of air every second at the aircraft’s speed V₀ and throws it out of the back at a jet speed V₉, the push on the engine is

F = ṁ × (V~9~ − V~0~)

plus a small correction for the fuel’s own mass and for any pressure left in the jet. That’s it. Everything else in this page is about how to make V₉ bigger than V₀, and how to do it without wasting fuel.

There are two obvious ways to get more thrust: accelerate a little air a lot, or a lot of air a little. Hold on to that thought; it decides the shape of every modern airliner engine, and we’ll come back to it.

The parts, in order

Here is the simplest engine there is: a turbojet, on a test bed, standing still.

What to try: click each part of the engine in turn, from the intake to the nozzle, and read the station table underneath.Open full tool →

The dots are air. They come in on the left cool and blue, turn yellow as the compressor squeezes them, glow red through the combustor, and leave on the right still hot. The circled numbers are the station numbers every engine maker uses, so that "T₄" means the same thing at Rolls-Royce and at Pratt & Whitney:

0 — the undisturbed air in front of the engine.
2 — the face of the compressor, after the intake has slowed the air down.
3 — the compressor exit: the highest pressure in the engine.
4 — the turbine inlet, just after the combustor: the hottest point.
5 — the turbine exit.
9 — the nozzle exit, where the jet leaves.

Try this. Scroll down to the station table. With this little engine (pressure ratio 8, turbine inlet 1,200 K, 20 kg of air a second) the compressor takes air at 288 K and 101 kPa and delivers it at about 558 K and 794 kPa. Nobody added any heat there — that 270 K rise is purely the work of squeezing it, the same reason a bicycle pump gets warm. The combustor then takes it to 1,200 K at almost the same pressure (a few per cent is lost to turbulence). The turbine drops it back to about 968 K and 289 kPa, and the nozzle turns what is left into a jet.

What to notice. The turbine takes out roughly 230 K of temperature, almost exactly what the compressor put in (270 K of cold air needs a bit less hot gas to pay for it, because hot gas carries more energy per kelvin). That is the deal at the heart of a gas turbine: the turbine exists to drive the compressor. Whatever energy is left after paying that bill is the engine’s profit, and it leaves as the jet.

Click the compressor, then the turbine, and compare what the tool says about each.

The Brayton cycle

Strip the engine down to thermodynamics and it does four things to the air, over and over:

Compress it (0 → 3), adding work.
Heat it at roughly constant pressure (3 → 4), by burning fuel.
Expand it (4 → 9), through the turbine and then the nozzle, taking work out.
Reject the leftover heat to the atmosphere, as the hot jet mixes away behind the aircraft.

That loop is the Brayton cycle, and it is the same cycle as a power station’s gas turbine, just with the output taken as a jet instead of a spinning shaft. Here it is for an idealised turbojet cruising at Mach 0.8 and 10 km, with perfect components:

What to try: scroll to the T–s and P–v diagrams, then raise the overall pressure ratio from 10 to 30 and watch thermal efficiency.Open full tool →

On the T–s diagram (temperature against entropy) an ideal compressor and an ideal turbine are vertical lines — no losses means no entropy made. Heating from 3 to 4 is the long curve along the top; the dashed line along the bottom is heat thrown away to the atmosphere. The area enclosed is the useful work. On the P–v diagram the same loop looks like a cartoon of the engine: squeeze the volume down from 0 to 3, swell it at constant pressure from 3 to 4, and let it expand all the way out to 9.

Try this. Note the thermal efficiency, about 54 %. For the ideal cycle it depends on one thing only: how much the air is squeezed before it is heated, counting the free squeeze it gets from ramming into the intake at Mach 0.8. Put the pressure ratio up to 30, then 40. The efficiency climbs every time.

What to notice. Now look at the other two efficiencies. Thermal efficiency is how well the engine turns fuel into jet kinetic energy. Propulsive efficiency is how much of that kinetic energy actually becomes useful push on the aeroplane — and at 35 % it is poor, because this jet leaves at over 1,100 m/s while the aeroplane flies at 240. All the energy in a jet that is much faster than the aircraft goes into stirring the air behind it. Multiply the two and you get overall efficiency: here 19 %. Hold on to that too.

Switch Cycle to With losses. The compressor and turbine lines on the T–s diagram lean to the right, every efficiency drops, and the engine is suddenly a real one.

Why not just squeeze harder?

If more pressure ratio always helps efficiency, why did early engines stop at 5 or 8? Partly because building compressors is hard (more on that below), but there is a subtler reason, and the trade-study tab shows it.

What to try: compare the dashed (ideal) and solid (with losses) curves, then change the swept input to turbine inlet temperature.Open full tool →

The left-hand chart is specific thrust — thrust per kilogram of air per second, which decides how big an engine has to be for a given thrust. It rises steeply, peaks around a pressure ratio of 10 for this engine, and then falls away. The reason is the ceiling on turbine inlet temperature. Squeeze the air harder and it leaves the compressor hotter, so there is less room left to add heat before you hit the 1,200 K the turbine blades can survive. Less heat added means less energy for the jet.

The right-hand chart is TSFC, thrust-specific fuel consumption: grams of fuel per second per kilonewton of thrust, the airline’s favourite number. Lower is better. The ideal curve keeps falling forever; the real one flattens out and eventually turns up, because each extra stage of compression costs a little more in losses than it gains.

Try this. Switch the swept input to Turbine inlet temperature. Specific thrust rises steadily with it: hotter turbines mean smaller, lighter engines for the same thrust. This is why turbine blades are single crystals of nickel superalloy, threaded with cooling passages and coated in ceramic, running in gas hotter than their own melting point. Half the history of jet engines is metallurgists buying another 50 K.

What to notice. On a turbojet, TSFC gets worse as the turbine gets hotter. More heat makes a faster jet, and a faster jet has worse propulsive efficiency. Keep that in mind for the turbofan.

The compressor: squeezing air uphill

A compressor is the hardest part of the engine to get right, because it is pushing air from low pressure to high — uphill, against the flow’s own wish to go backwards. A turbine has it easy: gas falling from high pressure to low pressure is happy to push blades round. That asymmetry is why a modern engine might have ten or more compressor stages driven by just one or two turbine stages.

What to try: drag the tip of V1 or W2 in the velocity triangles and watch the de Haller number; then scroll to the compressor map and lower the operating speed.Open full tool →

The top table estimates how many stages a compressor needs for the core engine’s pressure ratio, using different rules of thumb. Older subsonic stages managed a pressure ratio of about 1.15 each; transonic front stages manage 1.4 or more. Because the ratios multiply, a few better stages save a lot of length and weight.

The velocity triangles show what one stage does. The rotor blade moves at speed U; the air arrives with velocity V1, which the blade sees as W1 relative to itself. The blade turns the relative flow, the air leaves with more swirl, and that swirl — handed on to a stationary row of stator vanes and slowed down — becomes pressure. The work done per kilogram is simply U times the change in swirl.

Try this. Drag the tip of W2 to turn the flow more. Stage work and stage pressure ratio go up. Keep going until the de Haller number goes below 0.72. That is the warning that the relative flow is being slowed down too much: the boundary layer on the blade can’t cope and separates, the blade stalls like an aeroplane wing at too high an angle, and the stage stops pumping.

Now scroll to the compressor map. Pressure ratio is up the side, mass flow along the bottom, and the curved grey lines are speeds. The red line on the top-left is the surge line: try to make more pressure than that at too little flow and the whole compressor stalls at once. The high-pressure air rushes back out of the front, often with a bang and a flame out of the intake. The orange running line is where this simple engine actually operates at each speed.

What to notice. Pull the operating speed down. The surge margin changes as the point slides down the running line. Slam the throttle open and the combustor gets more fuel before the spool has sped up: the operating point jumps upwards, towards surge. That is why engines have bleed valves, variable stator vanes and fuel controls that are deliberately slow to answer the throttle.

Combustion: where the energy comes from

Between stations 3 and 4, fuel is sprayed into the compressed air and burned. The combustor is remarkable mostly for what it doesn’t do: it has to keep a stable flame in air rushing through at tens of metres a second, without letting the flame touch the walls or the turbine, and it uses only about a quarter of the air for burning. The rest goes round the outside of the flame tube and is mixed back in to cool the gas down to something the turbine can survive. Overall, a jet engine runs very lean: a fuel–air ratio around 0.02, where a car engine runs near 0.068.

The energy all comes from chemistry. Kerosene isn’t in the thermochemistry tool’s table, but natural gas is, and natural gas is what the gas turbines in power stations actually burn:

What to try: change H2O to H2O(g) and see how much less heat you get when the water leaves as steam.Open full tool →

Try this. The tool says burning one mole of methane releases about 891 kJ. Now edit the equation so the water is H2O(g) and the figure drops to about 803 kJ. The difference is the latent heat of the water. In an engine the water leaves as steam in the exhaust, so engineers use the lower figure — the lower heating value. Divide 803 kJ by methane’s 16 g per mole and you get about 50 MJ/kg. Kerosene gives about 43 MJ/kg, and that 43 is built into the jet engine tool’s burner sums.

What to notice. A modern airliner engine at cruise turns only around a third of that 43 MJ/kg into useful work on the aircraft. Where does the rest go? Mostly out of the back as heat, the dashed line on the T–s diagram — the price the second law of thermodynamics charges any heat engine.

The turbine and the nozzle

After the combustor, the gas passes through the turbine, which takes out just enough energy to drive the compressor (and in a turbofan, the fan). In our test-bed turbojet that left about 289 kPa and 968 K at the turbine exit — gas that is still under pressure and still hot, which is exactly what the nozzle needs.

The nozzle turns pressure into speed. A simple convergent nozzle like the one on our test-bed engine can only accelerate the gas to the speed of sound; past that the flow chokes. That’s what the tool reports on its Cycle tab — a core jet of 563 m/s at Mach 1.00, "choked (pressure thrust)" — with the gas leaving at about 153 kPa, well above the 101 kPa around it. The extra pressure pushing on the nozzle exit area is counted as pressure thrust. A convergent–divergent nozzle (the "Con-di" option on the Cycle tab) expands the gas fully and turns that pressure into extra velocity instead, which is why supersonic aircraft have nozzles whose shape changes in flight.

Another trick in the nozzle is the afterburner, or reheat: spraying fuel into the hot exhaust after the turbine and burning it again. With no turbine downstream to melt, the gas can go far hotter. Thrust jumps by something like half, and fuel consumption by far more, which is why fighters use it for take-off and combat and not to get anywhere. Open the afterburning turbojet on the cutaway to see where it sits: a second burner between the turbine exit (5) and the nozzle (7 to 9).

A lot of air, a little faster: the turbofan

Back to the choice we put off: accelerate a little air a lot, or a lot of air a little. Our turbojet chose the first, and its propulsive efficiency suffered. A turbofan does the second. It puts a large fan at the front, driven by an extra turbine, and blows most of the air around the core in a bypass duct, accelerating it only gently.

What to try: compare the size of the bypass stream (stations 13 and 19) with the core, then open the Cycle tab and read the propulsive efficiency.Open full tool →

This is a modern geared turbofan at cruise: for every kilogram of air that goes through the core, twelve go around it. That is the bypass ratio. The bypass stream has its own station numbers, 13 after the fan and 19 at its own nozzle, and you can see the blue, cool bypass air flowing round the outside while the core burns down the middle.

Try this. Open the Cycle tab. Propulsive efficiency is now about 70 %, against about 40 % for our little turbojet flying at the same speed and height. Overall efficiency roughly doubles, from around 15 % to around 31 %, and TSFC falls from about 35 to 17 g/(kN·s). Half the fuel for the same thrust — that’s the whole story of airliner engines since the 1960s.

What to notice. Specific thrust has collapsed, to about 113 N·s/kg. The engine moves so much air so gently that it needs an enormous fan to make useful thrust, which is why the engines on a modern twin look like they barely fit under the wing. Switch the engine to Turbofan (mixed exhaust) to see the low-bypass fighter arrangement, where the two streams merge before a single nozzle.

How far can you push it? The trade study answers:

What to try: follow the solid TSFC curve as the bypass ratio rises past 12, then sweep the fan pressure ratio instead.Open full tool →

As bypass ratio rises, TSFC falls and so does specific thrust — the engine gets more economical and bigger. Keep going and something odd happens around a bypass ratio of 12: the curves kink. The core can only supply so much power, and past that point the fan is asking for more than the turbine can give at this fan pressure ratio. The tool’s note says it plainly: "Past a point the fan asks more than the core can give."

Try this. Sweep Fan pressure ratio instead. A lower fan pressure ratio means a gentler, more efficient bypass jet, but you need even more air (and an even bigger fan) for the same thrust.

What to notice. The fan tip can’t go much faster than the speed of sound, and a huge fan has to turn slowly while the turbine that drives it wants to spin fast. Squaring that is why the newest engines put a gearbox between the fan and its turbine. The limit after that is the aircraft itself: a fan four metres across has to fit under a wing and clear the ground. Open rotors, with no duct at all, are the next step in the same direction.

The same machine on a car: the turbocharger

A turbocharger is a gas turbine with one important substitution: the car’s engine is the combustor. Exhaust gas from the cylinders spins a small turbine, the turbine spins a compressor on the same shaft, and the compressor pushes extra air into the cylinders, so they can burn more fuel per stroke. It has stations, a pressure ratio, a turbine paying for a compressor, and the same kind of compressor map with the same surge line.

What to try: find the red points left of the surge line, then raise 'Boost starts' and 'Full boost from' until they move clear.Open full tool →

This is a 2-litre four-cylinder engine with a compressor that is too big for it. Each dot on the map is the engine at full throttle at one rpm, from where boost begins to peak power. At high rpm the points sit in the middle of the efficiency islands, which is where you want them. But while the boost is building, between about 1,800 and 3,000 rpm, the engine isn’t swallowing much air yet and the compressor is being asked for a high pressure ratio at a low flow — up and to the left, past the surge line. The tool flags it in red.

Try this. Raise Boost starts and Full boost from so the boost arrives later. The points slide right and clear the surge line. Or lower the target boost. Either way you’re doing what a jet engine’s fuel control does when it refuses to let you slam the throttle open: keeping the compressor on the safe side of the map.

What to notice. Look at the compressor outlet temperature, about 135 °C at a pressure ratio of 2.3. That’s the same heat of compression as station 3 in the turbojet, and it is why turbo cars have an intercooler, to cool the air back down before it reaches the cylinders. The Turbine & wastegate tab balances the turbine against the compressor, exactly as the jet engine balances station 4 to 5 against 2 to 3.

What we’ve seen

Follow a parcel of air through a jet engine and every step makes sense. The compressor squeezes it and heats it up doing so. The combustor heats it more. The turbine takes back just enough to keep the compressor turning, and the nozzle turns the rest into speed. The Brayton cycle says squeezing harder makes the engine more efficient, and hot turbines make it smaller. Propulsive efficiency says that the best way to push an aeroplane is a lot of air, a little faster. That last idea grew the fan from nothing to four metres across.

Where to go next

Jet engines and gas turbines: the full tool. Try the Real engines tab, which loads approximate figures for real engine families into the cycle, and Model jets, for the fuel and run time of a KJ66-class model turbine.
A turboprop and a ramjet at Mach 3: the same cycle with the output taken as a propeller, or with no compressor at all because the speed does the squeezing.
Turbocharger matching: the full tool, with fuel, the turbine side and the intercooler.
Thermochemistry: heats of combustion, and why some fuels give more energy per kilogram than others.
Model rocketry: thrust without an intake. A rocket carries its oxidiser, so there is no compressor, no turbine and no ram drag.
E6B pilot calculators: the standard atmosphere, true airspeed and Mach number behind the flight condition.
Fluids: Reynolds number, drag and pipe flow, for the air around and through the engine.

How the Amiga drew Boing

2026-10-30T00:00:00Z

In January 1984, on a prototype that was still mostly breadboards, Commodore’s soon-to-be Amiga team showed a checkered ball bouncing across a grid. It spun as it went, it had a shadow, and every time it hit a wall the room went boom. Personal computers of the day could barely scroll a screen of text smoothly. This one was throwing a big, colourful object around at full frame rate while the processor, by most accounts, had very little to do.

That last part is the interesting bit. Boing was not a triumph of raw speed — the 68000 at the heart of the machine ran at about 7 MHz. It was a demonstration of delegation: a set of custom chips each doing one job, very fast, in step with the television beam. This exploration takes the demo apart one chip at a time. Every tool below is live; poke it as you read.

The thing we are taking apart

Here is a modern reconstruction, computed per pixel on your graphics card. It is cheating, in a sense — a GPU does billions of calculations a second — but it gives us the target.

What to try: drag the uSpin slider to 0, then to −3. Drag uBounce up to 2.Open full tool →

Try this. Set uSpin to zero and the ball slides rather than rolls; set it negative and it spins the wrong way, which looks instantly wrong to the eye. Push uBounce up and watch the arcs: the motion is just abs(sin(t)), a string of parabola-like hops with a sharp kiss at the floor.

Notice the four ingredients, because each one maps onto a different part of the real Amiga:

a background — the grid and the floor, which never changes;
a ball that moves around on top of it;
a spin that makes the ball look like a rolling sphere;
a shadow, and (not here) a sound.

The shader recomputes every pixel, every frame. The Amiga could not afford anything like that. Its trick was to avoid redrawing almost everything.

The beam, and the coprocessor that rides it

A 1980s television draws its picture with an electron beam, left to right, top to bottom, fifty times a second in PAL countries and sixty in NTSC ones. The Amiga’s display hardware is built around that beam. It fetches bitplane data just in time for the beam to paint it, and it keeps a little coprocessor, the copper, watching where the beam is.

The copper has only three instructions. WAIT stops it until the beam reaches a given line and horizontal position. MOVE writes a value into one of the custom chips' registers. SKIP jumps over the next instruction if the beam is already past a point. That is all — and it is enough to change the hardware’s mind halfway down the screen, every frame, without the processor lifting a finger.

What to try: press Play, then change a WAIT line or a colour value and watch the picture follow.Open full tool →

Try this. Press Play: the orange bar rides up and down because its WAIT line is an expression of the frame number t. Change the first $00A to $A0A and the band above the bar turns purple. Move over the picture to read the beam position, and click to copy the WAIT for that spot. Then pick Bouncing raster bars from the preset menu for the classic demo-scene effect.

Notice the line under the editor: fourteen instructions cost 56 bytes of chip memory. Every colour on this picture is the result of the copper writing one 16-bit register, COLOR00, at the right moment. Nothing in memory looks like a picture; the picture exists only as a series of timed register writes. Boing’s purple grid and its grey floor were an ordinary bitmap, drawn once, but the copper is still what loads the palette at the top of every frame — the operating system builds a copper list from each screen’s colour table. That matters for the next trick.

The spin is a lie: colour cycling

Rotating a checkered sphere properly means redrawing every pixel of it each frame. On a 7 MHz processor, for a ball that size, that was out of the question. So Boing’s ball was drawn once. Each checker on it was painted with a different palette index, arranged so that, going round the ball, the indices repeat in sequence. To make it spin, you do not touch the ball at all: you change which of those indices are red and which are white. Shift the red-and-white pattern along the palette by one entry each frame and the stripes appear to march round the sphere.

This is colour cycling, and the copper makes it nearly free, because the palette is loaded from the copper list every frame anyway. Here is the idea reduced to its bones: sixteen bands, each given its colour by a single MOVE, with the pattern shifting every eight frames.

What to try: press Play, then change t>>3 to t>>1 in a few lines, or swap rgb(15,1,1) for rgb(0,8,15).Open full tool →

Try this. Press Play and watch the stripes march down the screen. Nothing is moving in memory: each band is still at the line its WAIT names. Change t>>3 to t>>1 in one line and that band runs four times faster than its neighbours, breaking the illusion. Change the colours to see how little it takes to restyle the whole thing.

Notice the cost: one register write per band per frame. Boing’s real ball worked through a handful of palette entries rather than copper bands, but the principle is the same one — move the colours, not the pixels. It is the reason the ball could spin smoothly while the processor did almost nothing. (The shader at the top cheats the other way: it really does recompute the sphere, and its spin is honest maths.)

Moving a ball: sprites or bobs?

The spin is free. The bounce is not, because the ball’s pixels genuinely have to appear in a different place each frame. The Amiga offers three ways to move a picture around: hardware sprites, which the display overlays on the screen without touching the bitmap at all; blitter objects (BOBs), which are copied into the bitmap; and moving or scrolling the playfield itself.

Sprites are the tempting option. The display chip fetches them on its own and lays them over the playfield, so moving one is just two register writes. Here is a small spinning ball done that way.

What to try: press Play under Preview, then change HSTART and VSTART and watch SPR0POS and SPR0CTL change.Open full tool →

Try this. Press Play in the preview to run the four frames. Change HSTART and VSTART and watch the register values on the right: a sprite’s whole position lives in two words, SPRxPOS and SPRxCTL. Untick Attached pair and the colours collapse from fifteen to three.

Notice the width. An Amiga sprite is sixteen pixels wide, full stop. There are eight of them, and an attached pair spends two to get fifteen colours instead of three. Boing’s ball was many times wider than sixteen pixels; building it from sprites would have used them all up and still not been enough. Sprites were for mouse pointers, bullets and small heroes. A big ball had to live in the bitplanes.

The blitter: copying shapes through a mask

The blitter (block image transferer) is the chip that copies rectangles of memory. It reads up to three sources — A, B and C — combines them bit by bit with any Boolean function you choose, and writes the result to D. It does this one 16-bit word at a time, much faster than the 68000 could manage with its own instructions.

A blitter object is a shape plus a mask: a one-bit silhouette that says which pixels belong to the shape. Here is a 32 × 32 Boing ball BOB, two bitplanes deep, with eight frames of spin.

What to try: press Play under Preview, then change Blit at x from 37 to 32 and watch BLTCON0 and BLTCON1.Open full tool →

Try this. Run the preview to see the frames. Then look at the set-up underneath: BLTCON0 is $5FCA. Change Blit at x to 32 and the leading 5 becomes 0 — the shift disappears because 32 is a multiple of 16. Change it to 47 and the shift is back.

Notice the CA at the end of BLTCON0. That byte is the minterm: the Boolean function the blitter applies. $CA is the famous cookie-cut, D = AB + aC — where the mask A is set, take the shape B; where it is not, keep the background C. And the shift is there because the blitter only deals in whole 16-bit words: to put a shape at pixel 37 it reads the words and barrel-shifts them five pixels on the way through.

Now watch the blitter do that cookie-cut, word by word.

What to try: press ◀ to step back through the blit, then tap the destination to move the shape.Open full tool →

Try this. Press the step buttons to walk through the blit one word at a time: green for an A read, blue for B, orange for C, red for a D write. Tap somewhere else on the destination to move the shape and watch the shift and the masks change. Change the minterm to $FA (A + C) and the ball becomes a solid silhouette punched into the background.

Notice the extra word on each row. Because of the shift, a shape that is two words wide spills into a third, so the blit is one word wider than the shape and the last-word mask ALWM is $0000 to stop rubbish creeping in. Notice too what a bouncing BOB has to do every frame: first put the background back where the ball was, then cut the ball in where it is. That is two blits per frame, per object — exactly the kind of work the Amiga was designed to hand to the blitter instead of the processor.

The shadow fits the same scheme. The simplest way to get a see-through shadow on an Amiga is a bitplane of its own: wherever its bit is set, the palette maps each background colour to a darker version of itself. The shadow is one more one-bit shape, moved with the ball.

Who gets the memory?

All of this — the display fetching bitplanes, the copper, sprites, the blitter, the disk and the sound — reads and writes the same chip RAM. The bus is shared out in fixed slots along each scan line, and the processor gets whatever is left.

What to try: raise Bitplanes from 4 to 5 and 6, then pick the hires example.Open full tool →

Try this. This is a low-resolution PAL screen with four bitplanes. Each square is one colour clock: R is memory refresh, A0–A3 the four audio channels, S0a… the eight sprites, P1–P4 bitplane fetches, and M slots free for the 68000. Raise Bitplanes to 5 and then 6, and watch the free slots inside the display disappear. Choose Hires 640 × 256, 4 planes and the processor is locked out of chip RAM for most of every visible line. Tick Blitter busy and see what it takes for itself.

Notice why Boing works the way it does. A low-resolution screen with a few bitplanes leaves room on the bus for the blitter to move the ball and for the processor to run the program. Every effect that could be done by the copper or by colour cycling — the grid, the floor, the spin — costs almost nothing here. Only the ball’s movement really spends memory bandwidth, and the blitter spends it far more efficiently than the processor could.

The boom: Paula

The last ingredient is the one people remember from the show floor. When the ball hits a wall there is a deep, rubbery boom, and it comes from the side the ball hit. The story goes that the sound was recorded by thumping a garage door; however it was made, it was a sample — a recording — played back by the Amiga’s sound chip, Paula.

Paula has four channels, each fed by DMA straight from chip RAM (you saw their slots, A0–A3, in the last tool). Each channel plays 8-bit samples at a rate set by its period: the colour clock divided by a number you write to a register.

What to try: change Period to 214 and to 856, then switch between PAL and NTSC.Open full tool →

Try this. Period 428 plays at 8,287 Hz on a PAL machine — the note ProTracker calls C-2. Halve the period to 214 and the rate doubles, an octave up; double it to 856 and it drops an octave. Switch the clock to NTSC and the same period plays very slightly faster, because the colour clock is different. Type a sample rate on the right, such as 22,050 Hz, to see which period comes closest.

Notice how cheap it is. Once a channel is pointed at a sample and given a period and a volume, it plays on its own: one DMA word per scan line, two samples per word, no processor at all. And the channels are wired hard left and right — 0 and 3 to one side, 1 and 2 to the other — so making the boom come from the wall the ball hit is a matter of choosing which channel to start.

Putting it back together

Here is Boing again, rebuilt from the parts:

The grid and floor are an ordinary bitmap drawn once, displayed by bitplane DMA, with the palette loaded every frame by the copper.
The spin is colour cycling: the ball is drawn once with a run of palette indices and only the colours change.
The bounce moves real pixels, so the ball lives in the bitplanes and is moved with blits through a mask, at a shift that puts it on any pixel.
The shadow is one more one-bit shape that darkens whatever lies under it.
The boom is a sample handed to one of Paula’s channels, left or right.

The processor’s job was small: work out where the ball goes next, set up a couple of blits, nudge the palette, and start a sound when the ball hits something. Everything else ran on its own, in step with the beam. That is the lesson of Boing, and of the Amiga generally: the impressive part was not how fast it could compute, but how much it did not have to.

Where to go next

Copper and blitter playground — write longer copper lists: rainbows, sky gradients, palette splits and the line-255 trick.
Copper gradients and raster bars — design gradients and shaded bars and export them as a copper list.
Amiga sprites and BOBs — draw your own sprites and BOBs and export the data words and blitter set-up.
Amiga blitter lab — minterms, area fill and line mode.
Amiga DMA slot timing — every colour clock of a scan line.
Paula audio, MOD tracker and player — write music the way Amiga games did.
Amiga emulator and Amiga dev bench — run real Amiga software, or write 68000 code and try these tricks for real.
Boing Lamp effect lab — the ball as a lamp of 128 LEDs; its screensaver bounces it over the grid.
The Amiga 40th exhibit and Amiga games — what I have been building on the machine since.

How web search works

2026-10-30T00:00:00Z

You type four words into a box and, a quarter of a second later, ten results come back from an index of tens of billions of pages. It feels like one operation. It is really a pipeline of fairly simple ideas, each of which is easy to understand on its own and slightly astonishing when you see how far it has been pushed.

I have spent much of my working life on search — enterprise search, content platforms, more Lucene indexes than I care to count — and the thing I most wish someone had shown me early on is that there is no magic in it. There is text processing, a big sorted dictionary, some careful arithmetic and a lot of engineering to make it fast. This exploration walks through those pieces in the order a document meets them, with the site’s own search tools embedded along the way. Everything runs in your browser; poke at it as you read.

Step one: turning text into terms

A search engine never searches text. It searches terms: normalised tokens that came out of an analyser. Before a page is indexed, and again when you type a query, the text goes through the same chain: character filters (strip HTML, map odd characters), a tokeniser that splits the stream into words, and a sequence of token filters that lower-case, drop very common words and cut words down to a stem. If the document and the query go through the same chain, "Running" in one and "runs" in the other can meet in the middle.

Here is Elasticsearch’s english analyser, rebuilt stage by stage.

The english analyser: tokeniser, possessive stemmer, lowercase, stop words, keyword marker and Porter stemmer, one stage at a time.Open full tool →

Try this. Scroll down to the token stream and read it top to bottom. The sample sentence is The QUICK brown foxes jumped over the lazy dog’s bone. Running runners ran!

Watch the first stemmer turn dog’s into dog — that is the possessive filter, and it runs before lower-casing.
The stop filter removes both the tokens, but notice that the positions jump from 0 to 1 and from 5 to 7. The gaps are kept on purpose, so that a phrase query later knows the words were not really adjacent.
The Porter stemmer gives fox, jump, run, runner — and lazi. Stems are not words; they only have to be consistent. It also leaves ran alone: Porter strips suffixes, it does not know irregular verbs.
Switch the analyser at the top to standard and the stems disappear. Now a search for "fox" would not match "foxes". Switch to keyword and the whole sentence becomes one term — exactly what you want for a product code or a tag, and useless for prose.

That last experiment is the first real lesson of search: most "the search engine is broken" complaints are analysis problems. The document had the word; the analyser turned it into something the query never produced.

Step two: the inverted index

Once text is terms, the engine builds an inverted index: for every term, the list of documents that contain it. That list is called a postings list, and each entry — a posting — records the document ID, how many times the term appears (the term frequency, tf) and, if phrase search is wanted, the positions.

The same tool shows the index the english analyser builds over four tiny documents.

Four documents, seven distinct terms: each with its document frequency, total frequency, BM25 idf and postings.Open full tool →

Try this.

Find fox. It appears in all four documents (df 4), including "brown foxes and quick dogs", because the stemmer folded foxes into fox. Its idf is tiny: a term in every document says almost nothing about which one you want.
Compare jump, lazi and over, each in one document with an idf more than ten times larger. Rare terms carry the signal.
Look at the Field lengths and norms table under the index. Lucene stores each field’s length in a single byte, exact up to 23 and approximate after that. We will need those lengths in a moment, when we score.
Edit the documents. Add a line that says "the the the the" and watch it contribute nothing at all — the stop filter has already thrown those words away.

That is the whole shape of the index. A query for quick fox becomes "fetch the postings for quick, fetch the postings for fox, and combine them". The interesting questions are how to store lists with billions of entries, and how to combine them without reading all of them.

Step three: making postings small

A web index is dominated by its postings. If you stored every document ID as a 32-bit integer you would spend four bytes per posting, and for a term like "the" (if you kept it) or "online" that is a great many bytes. The trick that makes postings small is to store the gaps between sorted IDs — the d-gaps — rather than the IDs themselves. Common terms have small gaps, and small numbers can be written in very few bits.

The postings lab builds an index from a synthetic collection of 5,000 documents whose word frequencies follow Zipf’s law, as real language does, and compresses it every way you might reasonably try.

Bits per posting under every codec, for docID gaps, term frequencies and positions, on 5,000 Zipf-distributed documents.Open full tool →

Try this.

Read the docIDs column. Uncompressed is 32 bits a posting; VByte (Lucene’s old VInt: seven bits a byte, the eighth says "more follows") gets it under 10; Golomb and Rice coding, which are close to optimal when documents are scattered at random, get it near 6. The whole index drops from 3.66 MB to around 550 KB.
Now look at Simple-9, FOR-128, PForDelta and Lucene’s PForUtil. They are a bit or two worse than Golomb per posting, and that is the point: they pack 128 values at a fixed bit width so that a CPU can unpack them with shifts and masks, often with SIMD, instead of decoding bit by bit. Search engines trade a few per cent of space for several times the decoding speed.
Term frequencies compress superbly with γ or Golomb (about 1.4 bits), because nearly every tf is 1.
Roaring bitmaps look poor here at 18 bits a posting. They store document IDs, not gaps, and shine on dense lists and set operations — filters, mostly, rather than scored text.
Scroll down, pick a single term, and look at its gaps written out bit by bit.

Step four: not reading the whole list

Compression makes postings small enough to keep in memory. It does not stop a query for three common words from touching millions of documents. The second trick is to avoid scoring documents that cannot make the top ten.

The idea behind MaxScore and WAND is pleasingly simple. Each term has an upper bound: the best score it could possibly contribute to any document. Keep a heap of the best k results found so far; its lowest score is the threshold θ. A document that only contains terms whose upper bounds add up to less than θ cannot get in, so there is no need to score it — or even to stop at it. Block-Max WAND refines this with a maximum per block of 128 postings, so it can skip whole blocks.

The top 5 by BM25 over 5,000 documents: exhaustive scoring against MaxScore, WAND and Block-Max WAND, then step by step.Open full tool →

Try this.

Look at the Docs scored column. Exhaustive term-at-a-time and document-at-a-time scoring both score 3,463 documents. MaxScore scores 53. WAND and Block-Max WAND land around a hundred. Every row has a tick in Same top-k: the pruning is exact, not an approximation.
Notice why: the query’s first term, le, is in 3,111 documents but its upper bound is only 0.404, while the rare vuzesevi can contribute 2.417. Once the heap fills with documents that contain the rare words, the common term on its own can never compete.
Raise k from 5 to 100. The threshold rises more slowly, so pruning helps less — one reason engines retrieve a modest candidate set cheaply and rerank it with something heavier.
Use Step by step with WAND and watch the pivot: the first document at which the running sum of upper bounds could beat θ. Everything before it is skipped.

Step five: scoring a match with BM25

Retrieval finds the documents; scoring puts them in order. For thirty years the workhorse has been BM25, and it is still the first-stage scorer in Lucene, Elasticsearch and OpenSearch. For each query term it multiplies an idf (how rare the term is) by a tf factor (how often it appears here), with two knobs:

k1 controls saturation: how quickly repeating a word stops helping.
b controls length normalisation: how much a long field is penalised for being long.

BM25 on six short documents: “Lucene” three times barely beats “Lucene” once.Open full tool →

Try this.

The preset searches for lucene in the short-versus-long set. "Lucene Lucene Lucene" scores 0.395 and plain "Lucene" scores 0.390. Three times the term frequency buys about one per cent. That is saturation, and it is why keyword stuffing stopped working.
Look at the Classic TF-IDF column on the right. Lucene’s old scorer, which grows with √tf, ranks the one-word title first by a wide margin. Two formulas, two different answers.
Drag k1 to 3. Repetition keeps paying for longer and the gap widens. Drag it to 0 and tf stops mattering at all.
Drag b to 1 (or open example=length-norm) and length normalisation is total: a one-word document is the "densest" possible match, and the two Lucene-only documents tie.
Open the Curves tab to see the shapes behind these numbers: the tf curve flattening towards k1 + 1, the length curve tilting with b, and idf falling as a term gets common.
Click a row for Lucene’s full explain output, term by term.

BM25 is rarely the last word now. Production systems add field boosts, freshness, popularity, link-based authority and, increasingly, a learned model or a vector similarity on top. The Hybrid tab of the same tool fuses BM25 with toy embeddings using reciprocal rank fusion, which is how many teams combine lexical and semantic search today. But nearly all of them still use something like BM25 to pick the first few thousand candidates, because it is cheap, explainable and works straight off the inverted index.

Step six: knowing whether the ranking is any good

Change k1, add a boost, swap in a reranker — is the result better? Opinions are free, so search teams measure. The classic method is a set of queries with relevance judgements: people grade how useful each document is for each query, and metrics turn a ranked list of grades into a number.

One ranking with grades 3, 2, 3, 0, 1, 2 and two relevant documents never retrieved: precision, recall, AP, RR, nDCG and ERR.Open full tool →

Try this.

Read the coloured squares: the grades of the six results in rank order, and two dashed ones for relevant documents the system missed. Precision at 6 is 0.83 (five of six are relevant), but recall is only 0.71 because of the two it never found.
nDCG@6 is 0.785. DCG rewards high grades near the top, discounting each rank logarithmically; dividing by the ideal ordering (3, 3, 3, 2, 2, 2) makes it a number between 0 and 1.
Swap the 0 and the 3 in the grades box and watch nDCG drop. Put the best document last and it drops further. Position matters more than presence.
Try the ERR vs nDCG button. ERR models a user who stops at the first good result, so one perfect answer on top is nearly all it cares about; nDCG keeps counting.
For comparing two systems properly, see the Compare tab in the full tool (example=pooled-trec): 25 queries, BM25 against a reranker, and four significance tests that tell you whether the difference is real or noise.

Judged collections are expensive, so large engines lean on behaviour as well: clicks, dwell time, and interleaving, which merges two rankings into one list and lets real users' clicks vote. The Interleaving tab simulates exactly that.

Step seven: one index, many machines

Everything so far would run happily on one machine for a few million documents. The web is not a few million documents. Search engines split the index into shards — usually by document, so each shard holds a complete inverted index for its slice of the collection — and every query goes to every shard. Each shard returns its own top results with their scores, and a coordinator merges them.

Elasticsearch does exactly this inside a cluster, and the simulator shows it message by message.

Query then fetch: scatter to one copy of every shard, merge the candidates, fetch only the winners — then deep pagination and a slow node.Open full tool →

Try this.

Press Start. The first step scatters a search for error to one copy of each of the four shards; each returns only the IDs and scores of its own top five. The coordinating node merges those twenty candidates and only then fetches the five winning documents. Fetching every candidate’s full text would be far more expensive.
The narration points out something subtle: each shard scores with its own term statistics, so the same document could get a different idf on a different shard. On a big, well-mixed index the differences wash out; on a tiny one they can reorder results.
Keep going to Deep pagination. Asking for results 61–70 makes every shard return its top 70. This is why search engines cap how deep you can page, and why search_after exists.
The last steps slow one node down. A search waits for its slowest shard, so one sick machine slows every query that touches it — until adaptive replica selection learns to route around it.
Open the Fan-out tab. With a per-shard p99 of 60 ms and a 1% chance of any shard being slow, a search that touches 20 shards meets at least one slow shard 18% of the time, and its p99 is 126 ms — twice the p99 of any single shard. Drag the shard count up to see why this is called the tail at scale.

Step eight: the whole search engine

A web-scale engine wraps that core in a great deal more: crawlers and a document stream, a real-time index for pages published minutes ago next to the big batch-built index, a result cache (a surprising share of queries are repeats), query understanding (spelling, synonyms, intent), and a tree of root and mixer servers that fan each query out to thousands of index leaves and merge the answers. Each shard is replicated, partly for safety and mostly for throughput.

The system design tool has that architecture as a simulation, with an experiment that makes one leaf slow.

A web-scale search engine under load: one replica of one index shard becomes five times slower.Open full tool →

Try this.

Let the run play. Before the change at 5 s, p99 sits around 115 ms. After it, one replica of one shard is five times slower and p99 roughly triples while the error rate stays near zero. Nothing is "down"; the system is just slow, which is the hardest kind of failure to notice.
Find the slow leaf in the diagram: it is the busiest box, while its peers look normal.
Try Slow leaf, blind routing in the experiments list. Without least-loaded routing the damage is worse; with it, work steers away from the sick replica. Hedged requests — send the same request to a second replica after a short delay and take whichever answers first — trim the remaining tail.
Try Result cache cold and watch the index shards take the full load the cache was absorbing.
Switch from Simulate to Design or Deep dives for the architecture laid out box by box, or to Interview card for the condensed version.

Putting it together

Follow one query through what you have just played with. The words you type go through the same analyser the documents did, becoming terms. The root server sends those terms to every shard. Each shard decodes compressed postings lists, block by block, and uses upper bounds to skip documents that cannot make its top k. The survivors are scored with BM25 — and, these days, rescored by something more expensive. The root merges each shard’s best, fetches titles and snippets for the winners and returns them. Somewhere else, a team is measuring last week’s change with nDCG and interleaving to decide whether it stays.

None of the individual steps is hard. What makes web search remarkable is that each of them has been squeezed for decades: a few bits per posting, a few hundred documents scored out of millions, a few milliseconds per shard, and a tail latency watched as closely as the average.

Where to go next

The full tools, each with far more than the embeds show:

Analyzer and inverted-index playground — build your own analyser, try synonym graphs and edge n-grams for autocomplete, and phrase queries with slop.
Postings lab — compression codecs bit by bit, AND/OR intersection with skip pointers and galloping search, and MaxScore, WAND and BMW step by step.
Relevance and scoring explainer — BM25 in full, multi-field queries, function scores for popularity and recency, and hybrid search.
Ranking metrics — load your own TREC runs and judgements, compare systems with significance tests, and simulate interleaving.
Elasticsearch, simulated — indexing, near-real-time refresh, segments and merges, node failure and master elections.
System design interviews — the web search engine as a 60-minute mock interview, with the estimator to size it.
Vector search — the other half of modern retrieval.

And from my own work: ViewScene, a desktop tool for looking inside real Lucene indexes, and building a full social network from scratch, which has search as one of its parts.

Why sourdough rises

2026-10-30T00:00:00Z

A sourdough starter is about the least promising-looking thing in a kitchen: a jar of grey, slightly boozy paste that you feed flour and water once a day. Yet a spoonful of it, mixed into a kilogram of dough and left alone for a day, will lift that dough into a tall loaf with an open, glossy crumb and a crackling crust. No packet of yeast, no baking powder. Just flour, water, salt, time — and a few billion passengers.

This page follows that rise from start to finish: who lives in the jar, why the clock matters so much less than the thermometer, what the water is doing, how gluten holds the gas, how an aliquot jar tells you when to stop, and what happens in the first twenty minutes in the oven. Along the way the site’s own bread tools are open in the page, so you can push the numbers around and see what changes.

Who lives in the jar

A mature starter is a stable little ecosystem of two kinds of microbe.

Wild yeasts — often Saccharomyces cerevisiae, the same species as baker’s yeast, or Kazachstania humilis — eat sugars and turn them into carbon dioxide and ethanol. The carbon dioxide is the gas that raises the bread. The ethanol mostly bakes off, though you can smell it in a hungry starter.

Lactic acid bacteria — Fructilactobacillus sanfranciscensis (the one named after the San Francisco bakeries where it was first described), Lactiplantibacillus plantarum and their relatives — eat the same sugars and make lactic and acetic acid. They outnumber the yeasts by something like a hundred to one. They are where the sour in sourdough comes from, and the acid they make is what keeps the jar safe: within a few hours of a feed the pH falls low enough that most spoilage organisms simply cannot compete.

The two get along better than you might expect. Flour contains enzymes, amylases, that slowly break starch into maltose, a sugar made of two glucose units. F. sanfranciscensis loves maltose; K. humilis cannot use it at all and lives on the glucose the bacteria leave behind. They are not competing for the same meal, so neither crowds the other out. That partnership is why a healthy starter can be kept for decades.

Feeding is just dilution. You take a little of the old culture — a crowded, acidic, exhausted population — and give it a lot of fresh food. The microbes wake up, multiply, and the jar rises as gas is trapped in the paste. When the food runs low the rise stalls and the top begins to fall: that is the peak, the moment the culture is most active and the best time to use it.

What to try: switch the ratio between 1:1:1, 1:5:5 and 1:10:10 and watch the peak time move.Open full tool →

This view opens on an overnight feed: 10 g of starter with 50 g of flour and 50 g of water, a 1:5:5 feed, at 21 °C. The tool estimates it will peak in about ten and a half hours.

Try this:

Click 1:1:1. The peak time roughly halves. With equal parts starter and food, there are already plenty of microbes, so they only need a couple of doublings to eat it all.
Click 1:10:10. Now the culture is diluted twenty-one times over, and it needs more doublings to catch up — and each doubling takes the same time, so the peak moves out by hours.
Change Starter sits at from 21 to 26 °C and back. Everything moves — a hint of what the next section is about.

What to notice: the relationship is logarithmic. Going from 1:1:1 to 1:2:2 adds less time than you might guess, because what matters is how many doublings the culture needs, and that is the logarithm of how much you diluted it. Bakers use big ratios on purpose: a 1:5:5 feed in the evening is ripe at breakfast, and a more diluted, slower culture tastes milder.

Temperature is the real clock

Microbes are chemistry in a bag, and chemistry speeds up with heat. For yeast and lactic acid bacteria in the range a kitchen actually covers — roughly 4 °C to 30 °C — a useful rule of thumb is that activity doubles for every 10 °C warmer. Scientists call this a Q₁₀ of about two. It is not exact, and it breaks down above about 32 °C (the bacteria start to outrun the yeast) and near the fridge (the yeast nearly stop), but over the range that matters it is surprisingly good.

What to try: drag your eye along each curve from 16 °C to 30 °C; then change Starter sits at and watch the dashed line move.Open full tool →

The chart plots hours to peak against the jar’s temperature for four common feeding ratios. The dashed line marks the temperature you typed into Starter sits at.

Try this:

Follow the 1:5:5 curve. At 16 °C it takes around fifteen hours; at 26 °C, about half that. Ten degrees, half the time.
Look at the gaps between the curves. At every temperature, diluting further adds a roughly constant stretch of time, because each extra doubling takes as long as the last.
Below the chart there is a form to calibrate the model to your starter: enter a feed you actually did, how long it took to peak and how warm it was. Every other estimate on the page re-scales from that one observation.

What to notice: the shape is the important part, not the numbers. Your starter, your flour and your kitchen will shift the curves up or down, but the curve stays a curve. A warm summer kitchen and a cold winter one can be ten degrees apart, and that is the difference between a bake that runs on time and one that runs half a day late. Experienced bakers watch the thermometer far more than the clock.

Water, flour and the baker’s percentage

Before the dough can rise it has to be mixed, and bakers write their recipes in a slightly odd way: every ingredient is a percentage of the flour. A dough with 1,000 g of flour and 750 g of water is "75 % hydration". It makes recipes easy to scale, and it makes the one number that most shapes a loaf — how wet it is — visible at a glance.

Water matters to the rise in three ways. It is the medium the microbes and enzymes work in, so a wetter dough ferments a little faster. It lets the gluten proteins move and link up. And it makes the dough more extensible, so gas cells can grow bigger before they tear: wetter doughs tend to give a more open crumb, at the cost of being harder to handle.

There is a catch with sourdough, though. The levain — the portion of starter built up for this particular bake — is itself flour and water. If you only count the water you poured in, you are underestimating how wet the dough really is.

What to try: scroll to Overall and compare its hydration with the 75 % on the water row; then change the levain to 30 %.Open full tool →

This is a country loaf in the Tartine style: 90 % bread flour, 10 % whole wheat, 75 % water, 2 % salt and 20 % levain at 100 % hydration, scaled to two 900 g loaves.

Try this:

Scroll down to Overall. The water row says 75 %, but the true hydration — all the water over all the flour, the levain’s included — is 77.3 %. About 9 % of the flour has already been fermented before the dough is even mixed.
Change the levain’s baker’s percentage from 20 to 30. The true hydration climbs again, and so does the prefermented flour: more of the dough arrives already full of microbes and acid.
Change the water to 68 %. That is closer to a sandwich loaf; 80 % and up is ciabatta territory.

What to notice: "prefermented flour" is a quiet but important number. It is how much of the dough has already been through fermentation — and so how much acid, flavour and active culture you are starting with. More levain means a faster bulk and a tangier loaf; less means a longer, gentler one.

Choosing the dough’s temperature

Because temperature sets the pace, bakers aim the dough at a target — usually 24–26 °C — by choosing the temperature of the water. Jeffrey Hamelman’s desired dough temperature method adds up the flour, the room, the water, the levain and the heat the mixer adds through friction; together they should come to four times the target. Solve for the water and you are done.

What to try: lower the room and flour temperatures from 30 °C to 20 °C and watch the ice disappear.Open full tool →

This preset is a hot kitchen: 30 °C room and flour, a stand mixer adding about 8 °C of friction, and a target of 24 °C. The water has to be 4 °C — colder than the tap — so the tool tells you to replace about 16 % of it with ice.

Try this:

Change Mixing to Hand mixing. Less friction, so the water can be warmer and the ice share shrinks.
Set room and flour to 18 °C, a winter kitchen. Now the water needs to be warm.

What to notice: mixing alone can warm a dough by ten degrees. Combined with the doubling rule, that is the difference between a four-hour and an eight-hour bulk — set before the first fold.

Gluten: the balloon around the gas

Yeast makes gas, but gas alone does not make bread; it would bubble straight out. What holds it is gluten. Wheat flour carries two families of storage proteins, glutenins and gliadins. Dry, they do nothing. Wet them and work them, and the glutenins link end to end into long elastic chains while the gliadins slip between them like a lubricant. The result is a network that is both springy (glutenin) and stretchy (gliadin) — a sheet that can inflate.

One surprising detail: the yeast do not make the bubbles. Mixing folds in tiny pockets of air, and every gas cell in the finished crumb grew from one of those. The carbon dioxide dissolves in the dough’s water and diffuses into the pockets that already exist. That is why mixing and folding matter so much: they decide how many cells there are and how strong their walls are. Stretch-and-folds and coil folds during the first half of bulk build strength without knocking out gas, and organise the network so the cells grow evenly.

Acid plays both sides. A little acid tightens the gluten and makes dough easier to handle. A lot of acid, and the flour’s own protease enzymes given many hours, start to break the network down again — which is why an over-fermented sourdough turns slack, sticky and impossible to shape. (Rye is a different story: it has very little useful gluten, and the acid in a rye sour is what lets the crumb set at all, by slowing the enzymes that would otherwise turn the starch to paste.)

Knowing when bulk is done: the aliquot jar

Bulk fermentation — the long first rise, after mixing and before shaping — is where most sourdough goes wrong. Stop too early and the loaf is tight and dense with a few big tunnels under the crust. Go too far and the gluten weakens, the dough spreads, and it bakes flat and sour. The clock is a poor guide because, as we have seen, it depends so heavily on temperature and levain.

The trick many bakers use is the aliquot jar: right after mixing, drop a small piece of the dough into a straight-sided jar, push it flat and mark the level. The jar sits beside the main dough, ferments at the same rate, and its straight sides turn a vague "looks puffy" into a percentage you can read.

What to try: scroll down to Fermentation model; change the dough temperature and the levain percentage and watch the bulk time and the aliquot target move.Open full tool →

The tab opens on a guide to starter types — liquid levain, stiff levain, rye sour, lievito madre — which is worth a read in itself: note how a stiff, cooler levain favours acetic acid and a sharper loaf, while a liquid, warmer one favours milder lactic acid. Scroll down to Fermentation model. The preset is 15 % levain in a 26 °C dough: about four and a half hours of bulk, and an aliquot-jar range to aim for before shaping.

Try this:

Look at the table of bulk times. Read across a row (warmer) and down a column (more levain). The fastest corner — 30 % levain at 28 °C — is under three hours; the slowest — 5 % at 20 °C — is nearly ten.
Change the dough temperature to 22 °C. The bulk time stretches, and the aliquot target rises. A cooler dough ferments more evenly and can be taken further.
Tick and untick Cold retard after bulk. If the shaped loaf is going to spend the night in the fridge, it keeps fermenting slowly there, so bulk should stop earlier.

What to notice: the target is a range, not a line, and it moves with the conditions. The tool says so plainly: these are rules of thumb to calibrate against your own crumb. Bake, cut the loaf open, and adjust next time.

When things go wrong

Every baker has pulled out a pale, flat disc with no ear. The useful thing about knowing the mechanism is that a bad loaf becomes evidence: the crumb tells you which part of the chain failed.

What to try: read the ranked causes, then untick Spread flat and tick Dense instead.Open full tool →

This preset ticks three symptoms — spread flat, no ear, a tight even crumb — and ranks the likely causes beneath.

Try this:

Read the list. Over-proofing comes first, then weak shaping, under-developed gluten, too little steam, a weak starter or a cold dough, and timid scoring. Each comes with what to change.
Untick Spread flat and tick Dense, heavy crumb and Big holes or a tunnel under the top crust. The ranking flips towards under-fermentation: the dough simply did not have enough gas or time.
Scroll down to Reading the crumb for drawings of what well-fermented, tight, dense, gummy and under-proofed crumbs look like.

What to notice: the causes map neatly onto the sections of this page — the microbes, the temperature, the gluten, and (next) the oven. Change one thing at a time between bakes, or you will never know which one fixed it.

Shaping, scoring and the oven

After bulk the dough is divided, pre-shaped, rested and shaped. Shaping is not just cosmetic: it drags the outside of the dough into a taut skin that holds the loaf up while it proofs and steers its expansion in the oven. Then comes the final proof — an hour or two on the bench, or a night in the fridge, which slows fermentation to a crawl, deepens the acid flavour and makes a cold, firm loaf that is easy to score.

What to try: step through the bâtard drawings, then scroll to Scoring and compare the single ear with the cross.Open full tool →

This view shows shaping a bâtard in four steps, the poke test for judging the proof, and the scoring patterns, opened on the single ear.

Try this:

Read the poke test: a dent that springs straight back needs more time; one that springs back slowly and only part way is ready; one that stays is over-proofed.
In Scoring, compare the single ear (a shallow 30–45° cut that lifts a flap of skin) with the cross or the square on a boule. Each cut is a deliberate weak point: it decides where the loaf will burst open as it springs.

When the loaf hits a hot oven, the rise has one last act — oven spring. In the first ten minutes or so the dough warms from the outside in. Gas expands with heat; the dissolved carbon dioxide comes out of solution; ethanol and then water turn to vapour inside the cells; and the yeast have a final frantic burst of activity before they die at around 60 °C. A well-made loaf can grow by a third in that time. Then the starch gelatinises and the proteins set, freezing the crumb in place, and the crust dries and browns through the Maillard reaction and caramelisation.

All of that depends on the crust staying soft long enough. That is what steam is for.

What to try: switch between the Dutch oven, the stone with steam and the baking sheet, and compare the steam phase.Open full tool →

The country loaf is set up for a Dutch oven: preheat for 50 minutes at 250 °C, bake 20 minutes with the lid on, then 25 uncovered at 230 °C until the middle reads 96–99 °C.

Try this:

Switch the set-up to Stone or steel + steam, then to Baking sheet + cover. The steam phase changes shape, but it is always there.
Read Why steam, and when it is done. Moist air keeps the skin supple so the loaf can expand and the score can open; condensing steam gelatinises the surface starch for a glossy, crackly crust. After that, dry heat browns it.

What to notice: the lid-on phase is exactly the window of oven spring. The Dutch oven works because the loaf makes its own steam and the lid traps it. Take the lid off too early and the crust sets while the loaf is still trying to grow: a pale, tight, earless loaf, just like the one in the troubleshooter.

And then — the hardest step — leave it to cool. The crumb is still setting as the steam leaves it. Cut a lean loaf before an hour is up and it will be gummy; cut a high-rye loaf before a day is up and it will be worse.

Putting it together

Follow one loaf through and the whole story fits in a paragraph. A feed dilutes a culture of yeasts and lactic acid bacteria and gives them maltose to share; they double until the food runs out, and the jar peaks. That ripe culture goes into a dough whose water you chose to land it at 25 °C, because temperature sets the pace of everything that follows. Mixing and folding make the air cells and build the gluten network around them; the yeast fill the cells with carbon dioxide while the bacteria make the acids that flavour the loaf and, given too long, weaken it. The aliquot jar tells you when to stop. Shaping makes a skin, scoring makes a weak point, and in a steamy oven the gas expands one last time before heat sets it for good.

None of it needs a packet of yeast. It needs a thermometer, a straight-sided jar and patience.

Where to go next

Sourdough starter and levain planner — feeding, building a levain that lands exactly on its hydration, peak times calibrated to your starter, and a bake-day schedule you can export to a calendar.
Bread and baker’s percentages — write any formula, see its true hydration, convert yeast and work out the water temperature.
Artisan bread — about ninety breads with their process, a bake-day plan with fold timers and the aliquot-jar target, shaping and scoring drawings, steam set-ups and a troubleshooter.
Fermentation salt and timing — the same lactic acid bacteria, put to work on sauerkraut and pickles instead of bread.
Pizza dough — a sourdough pizza is the quickest way to practise all of this in an evening.

3-D models

2026-10-18T00:00:00Z

Open any model to see it from every side — front, side, top and the print plate — and the ones marked 3-D you can turn round in your browser, or stand on your desk in AR on a phone. The chips pick out one project at a time.

How the Lemmings came about — friends from the Amiga community, designed with Meshy from photos and printed on Bambu printers — is in Lemmings, Lemmings everywhere. This page grows as I add models; the date at the top says when the last one arrived.

93 models in 12 projects

AI Force

2026-10-18T00:00:00Z

The administration is starting a new military force, and it got me wondering what an AI Force would actually look like. Not the sleek recruiting-poster version — the real one. So I made a squad of them in Meshy, and they look like the people who would really sign up: typical geeks, squeezed into black hero suits with glowing circuit lines.

The Pudgy Guyprinted 100 × 71 × 44 mmDrag to turn · pinch to zoom · on a phone, view it in your room

The squad

Big Brain — the strategist. Most of him is head.
Neckbeard — arms crossed, in two uniforms: the hero suit, and the blue polo and chinos he wears the rest of the time.
Pudgy Guy — glasses on, running. Probably towards the snacks.
Python Woman — curly hair, flexing a bicep. She wrote the code everyone else depends on.
The Teen — grinning, fist in the air, already better at this than the rest of them.

Each one comes in two versions: Meshy’s first texture, and a re-textured, more realistic 4K one.

Python Womanprinted 100 × 62 × 26 mmDrag to turn · pinch to zoom · on a phone, view it in your room

Open any of them below to see it from every side.

12 models

A nod to Atari Force

The look is a homage to Atari Force, the comic DC Comics made for Atari in the 1980s: a crew in matching jumpsuits striking heroic poses. I made the original crew as well — the AI Force is what you get when you send the same poses to a hackathon.

Martin Champion

Lydia Perez

Dr Lucas Orion

Li-San O’Rourke

Mohandas Singh

The sixth crew member

All of these, and the rest of my models, are in 3-D models.

Aliens in my house

2026-10-18T00:00:00Z

I’m a big fan of the alien genre, and I wanted some funny aliens around the house. So I made a little crowd of them in Meshy, printed them, and painted them by hand with airbrush "felt tip pens" (yes, that is a thing).

The true take on see no evil, speak no evil and hear no evil.

The morning crew

A nice cup of Joe in the morning provides clarity for today’s probing.

The coffee alien — drag to turn itprinted 45 × 44 × 100 mmDrag to turn · pinch to zoom · on a phone, view it in your room

Back off… aliens need space too!

E.T. phone home.

Not all of them are friendly

Pan-galactic greeting.

See no evil

The trio on my shelf, as models:

See no evilprinted 100 × 32 × 33 mmDrag to turn · pinch to zoom · on a phone, view it in your room

All my aliens — including the park bench they like to sit on — are below; open one to see it from every side.

12 models

Space: 1999

2026-10-18T00:00:00Z

Space: 1999 was one of the shows I loved to watch as a kid: the Moon blown out of Earth’s orbit, the crew of Moonbase Alpha along for the ride, and the Eagle Transporters — the best spaceships on television. So when I started making 3-D models, a couple of them had to come from it.

The Eagle Transporter

The Eagle Transporter — drag to turn itprinted 100 × 46 × 20 mmDrag to turn · pinch to zoom · on a phone, view it in your room

I made the Eagle twice. This one is a single piece, made from one picture. The other version was made from several pictures, and prints as five parts on one plate: the hull and the four leg pods.

A Moonbase Alpha astronaut

An astronaut in the orange suit and yellow helmetprinted 33 × 27 × 100 mmDrag to turn · pinch to zoom · on a phone, view it in your room

The astronaut is in the show’s orange spacesuit and yellow helmet. This version was made from several pictures, so the backpack is modelled too; the first version was made from one.

The rest of my spacecraft — an Apollo spacesuit and Blake’s 7’s Liberator among them — are with everything else in 3-D models.

gitwebsite is now open source

2026-10-12T00:00:00Z

gitwebsite is the tool that builds and publishes this site. Until today the engine and my site’s content lived in one private repository. I’ve split them: the engine is now public at github.com/geekychris/gitwebsite under the MIT licence, and my own site’s source stays in a separate private repository.

What’s in it

A static-site engine in Rust. Content is AsciiDoc with a short header (type, date, areas, tags, summary). Items sit in a tree of areas you define, so one post can belong to several.
gw, a command-line tool, for creating, checking, building, previewing, and publishing.
A REST API and a desktop app (Tauri) with a management UI and a live preview. The app has a 🎤 button beside every text box for macOS dictation.
An MCP server, so an AI assistant can work with the site through the same operations — plus a set of Claude Code skills for writing, curating, adding media, and publishing.
A registry, ideas list, and media library: where each piece of work came from, what to write next, and images (including SmugMug-hosted photos).
Publishing to GitHub Pages, leaving other folders in the Pages repository alone.

Figure 1. The app: editing a post beside a live preview (on a demo site)

Writing with Claude

Two workflows use the MCP server and the app together:

Project stories. You drop in photos, choose the ones that matter with a comment each, and tell the story roughly; Claude interviews you a few questions at a time and drafts the post. The questions appear in the app, so you can answer there.
Enhance with Claude. Open an existing article or project and press Enhance with Claude. You answer whichever key questions you like — the inspiration, why it exists, the hard parts, what’s next — and Claude reads the post and the code repository behind it, asks follow-ups, and writes a proposed revision. The app shows it as a diff against the current post; you accept it, ask for changes, or discard it. Nothing is published until you publish.

Figure 2. Enhance with Claude: a question waiting on the page, and the proposal as a diff

Using it for your own site

gw init creates a new site with a default theme and starter content, ready to check, build, and publish. A settings file keeps a list of sites, so the app can switch between several. The repository’s README covers building from source, setting up GitHub Pages, and installing the app and the Claude Code skills.

Some things here are specific to my site and aren’t part of the engine — the browser tools, for example, live in my site’s own repository. The projects page grouped by area (Software, Retro computing, …) is an engine feature you can configure in site.yaml.

Figure 3. The default theme’s projects page, grouped by area

AOEE: a TAO-style social graph cache in Rust

2026-10-10T00:00:00Z

Every social product eventually asks the same handful of questions, millions of times a second. Who does this person follow? Do these two people know each other? Which friends do we have in common? Who liked this post, and how? Who might this person want to meet next?

None of those questions is hard on its own. What makes them hard is the shape of the data: a few accounts with millions of edges, millions of accounts with a few dozen, and a workload that is overwhelmingly reads. Asking a relational database to answer "mutual friends" with a self-join on an edges table works right up until it doesn’t.

Facebook’s answer, described in the 2013 USENIX paper on TAO, was to treat the graph as a cache of associations in front of MySQL and to serve the read patterns directly. AOEE — Attribute Object Enterprise Edition — is my take on that idea, built as the graph layer for the enterprise social platform I was writing about in another article. The code is on GitHub.

Figure 1. How a request reaches the graph: the JVM side handles HTTP, datasets, and benchmarks; the graph lives in a sharded Rust server behind gRPC.

The data model: ids, edge types, and edge keys

Everything in AOEE is a 64-bit EntityId. The top 16 bits hold the entity type (user, post, comment, photo, group, page, event, tag, or a custom type) and the bottom 48 bits the id itself. Knowing what an id refers to costs a shift, not a lookup, and ids stay plain integers — which matters a great deal for compression, as we’ll see.

Ids can also be generated without a coordinator: the 48-bit part becomes a 32-bit timestamp (seconds since a 2020 epoch) followed by a 16-bit sequence. Any client in any language can mint them.

Relationships are typed, and almost every type comes as a forward/reverse pair: FOLLOWS/FOLLOWED_BY, LIKES/LIKED_BY, AUTHORED/AUTHORED_BY, MEMBER_OF/ HAS_MEMBER, TAGGED_IN/HAS_TAG, and so on. FRIEND_OF is its own reverse. The clients know each type’s reverse, so an application stores both directions and "who follows Alice?" is as cheap as "whom does Alice follow?" — no reverse index to scan. A few types carry a one-byte metadata value per edge — LIKES uses it for the reaction (like, love, haha, wow, sad, angry).

The unit of storage is the posting list for an EdgeKey — a source entity plus an edge type. EdgeKey(User:42, FOLLOWS) is the sorted list of every entity user 42 follows.

That word posting list is deliberate. Most of my career has been in search, and a search engine’s inverted index is exactly this: for each term, a sorted list of document ids, compressed, with fast intersection. Swap "term" for "user and relationship" and the whole toolbox comes across — delta encoding, variable-length integers, skip tables, galloping intersection. AOEE is, in a sense, a search engine whose documents are people.

Posting lists: an LSM tree in miniature

Graphs are mostly read, but they are written constantly, and a perfectly compressed sorted array is miserable to insert into. So each posting list is a tiny log-structured merge tree.

Figure 2. A posting list is an append-only write buffer plus immutable, sorted, encoded segments; compaction moves one into the other.

New edges and deletes go into an unsorted write buffer: the destination id, a nanosecond timestamp, the metadata byte, and a tombstone flag for deletes. When the buffer passes a threshold (256 entries by default in the current code), it is compacted: sorted, deduplicated, tombstones applied, and written out as an immutable segment with the smallest workable encoding. Small segments are merged into larger ones by leveled compaction, the same idea as in RocksDB and LevelDB.

Reads walk a merged iterator over the buffer and the segments, newest first. Each segment records its minimum and maximum id, so a contains check skips segments whose range cannot hold the target, and delta-encoded segments carry a skip table so a lookup can jump close to the answer and decode forward from there rather than decoding the whole segment.

Four encodings, chosen per segment

Social-graph lists span six orders of magnitude, and no single encoding suits all of them. AOEE picks one per segment:

Encoding	When	What it does
SmallVec	under 128 ids	A plain vector of 64-bit ids. For short lists, compression is not worth the CPU.
DeltaVarint	128 to 4,096	Store the first id, then the gap to each next id as an LEB128 varint. Gaps under 128 cost one byte.
BlockPacked	over 4,096	Fixed blocks of 128 ids, bit-packed to the width the block needs — compact, with good random access.
Roaring	over 100,000, or dense	Roaring bitmaps, which shine when ids cluster and make set operations fast.

Encoding

When

What it does

SmallVec

under 128 ids

A plain vector of 64-bit ids. For short lists, compression is not worth the CPU.

DeltaVarint

128 to 4,096

Store the first id, then the gap to each next id as an LEB128 varint. Gaps under 128 cost one byte.

BlockPacked

over 4,096

Fixed blocks of 128 ids, bit-packed to the width the block needs — compact, with good random access.

Roaring

over 100,000, or dense

Roaring bitmaps, which shine when ids cluster and make set operations fast.

Density is simply the count divided by the id range. The repo’s encoding notes work an example: a user with 1,000 followers whose ids run 5,000 to 5,999 takes 8,000 bytes as raw 64-bit integers and about 1,003 bytes delta-encoded — an 87% saving — because every gap is one byte. This is where keeping ids as integers pays off: the 48-bit layout keeps ids of the same type numerically close, so their deltas stay small.

The design point I like best is that the choice is invisible above the segment. Every encoding implements the same PostingIterator trait — next, peek, and seek to the first id at or above a target — so the set operations never know or care how a list is stored.

Set maths instead of joins

"Mutual friends" is an intersection of two sorted lists. "Liked by any of my friends" is a union. With every list sorted and seekable, both are linear merges — no hash tables, no joins, no temporary results.

The interesting case is lopsided input: a user with 80 friends intersected with a celebrity who has two million followers. Merging two million ids to find a handful of matches is wasteful. AOEE’s adaptive intersection checks the sizes, and when one list is more than ten times larger it switches to galloping: for each id in the small list, seek forward in the large one — a binary search over what remains, rather than a step-by-step merge. The cost grows with the small list and only logarithmically with the large one.

Intersection and union also generalise to many lists at once (intersect_many, union_many), which is how multi-way queries such as "groups we are both in that also contain this post’s author" stay cheap.

Friend-of-friend, with a budget

Friend suggestions are the classic two-hop query: for each of my friends, look at their friends, count how many times each person appears, drop me and the people I already know, and rank by that count — the number of mutual connections.

Done naively, one well-connected friend can blow the query up. The FofQuery executor has explicit limits:

a fan-out cap on how many first-hop friends are expanded (1,000 by default), with optional deterministic sampling when someone has more;
an optional time budget, after which it returns what it has and marks the result as truncated;
a minimum score and a maximum number of results;
exclusions for yourself, your existing connections, and anything else you pass in, such as blocked users.

The result reports how many neighbours were processed and whether it was truncated, so the caller can tell a complete answer from a best effort. I think that honesty matters more than the algorithm: a recommendation that is cheap and known to be partial beats one that is exact and occasionally takes two seconds.

Sharding and the server

The Rust server is a workspace of crates: aoee-core (ids, edge types, encodings, iterators, set operations, friend-of-friend), aoee-storage (an EdgeStore trait with an in-memory implementation), aoee-shard (shards and routing), aoee-server (the gRPC service), plus a Rust client and a benchmark crate.

Each EdgeKey maps to a shard through a consistent-hash ring with virtual nodes, so adding shards moves only a fraction of the keys. Within a shard, posting lists live in a DashMap — a concurrent hash map with fine-grained locking — so readers of different keys never contend.

The JVM side, the UI, and persistence

Most of the systems I work with are Java, so AOEE has a first-class Java client:

AoeeClient client = new AoeeClient("localhost", 50051);

client.addEdge(userId, EdgeType.FOLLOWS, targetId);
client.addEdge(userId, EdgeType.LIKES, postId, 0, ReactionType.LOVE.getValue());

List<Long> mutual = client.intersect(
    new EdgeKey(alice, EdgeType.FRIEND_OF),
    new EdgeKey(bob, EdgeType.FRIEND_OF));

FofResponse suggestions = client.friendOfFriend(alice, EdgeType.FRIEND_OF);

On top of it, a Spring Boot proxy exposes the same operations as REST, loads datasets, and runs benchmarks; a React UI provides a dashboard, a force-directed graph explorer, an edge manager with a reaction picker, a query builder for set operations and suggestions, and a dataset loader. An optional persistence service (REST and GraphQL over H2 or PostgreSQL) receives writes through the proxy so the cache can be warmed from a database, and both the server and the proxy have Dockerfiles. To run the whole stack locally:

./start-services.sh            # Rust server, Spring proxy, React UI
./start-services.sh --persist  # ...plus the persistence service

What the numbers say

The benchmark harness generates a synthetic social graph — users with a skewed follow distribution, posts, groups, likes with reactions, a few very popular accounts and a viral post — then times each operation. The small preset builds 1,000 users, 2,000 posts, and 10 groups: 125,657 edges.

In the run recorded in the repo’s benchmarking guide, median latencies for every operation — neighbour lists, lists with reaction metadata, contains, counts, mutual-friend intersections, friend-of-friend, the popular-user lists, and the viral post — fell between 64 and 77 microseconds, with 99th percentiles between 93 and 172 microseconds. The slowest single call was a friend-of-friend query at 1.9 milliseconds.

Two caveats, because benchmarks deserve them. Those numbers are measured end to end from the JVM through gRPC on a development machine, so most of each microsecond budget is transport, not graph work. And the throughput figures in that same output came from a calculation I later found was off by a factor of a thousand — fixed in the commit history, and a good reminder to sanity-check any number that looks either too good or too bad.

What I learned

Search engine techniques transfer beautifully to graphs. Posting lists, gap encoding, skip tables, and galloping intersection solved most of the problems I expected to fight.

Adaptive representation beats a clever single format. The four encodings are each simple; the win is choosing between them per segment and hiding the choice behind one iterator.

Put limits in the query, not around it. Fan-out caps, sampling, and time budgets made friend-of-friend predictable, and reporting truncation made it trustworthy.

Write paths want buffers; read paths want immutability. A small LSM per posting list let both sides be simple.

What’s next

The repo’s roadmap covers the obvious next steps for a production cache: a RocksDB-backed store, leader–follower replication, set operations that span nodes, streaming updates to the UI, a cache for hot friend-of-friend results, time-range queries (one place TAO is still ahead — AOEE orders lists by id, not time), and SIMD-accelerated intersection.

Amiga DevBench: a modern dev loop, and an AI pair programmer, for the Amiga

2026-10-10T00:00:00Z

Developing for the Amiga in 2026 is a strange mix of joy and friction. The machine is still wonderful: a pre-emptive multitasking OS in 512 KB of ROM, a custom chipset you can drive directly, and a community that has kept it alive for forty years. But the development experience has not kept up. There is no source-level debugger running on the target, no network stack on a stock machine, and one bad pointer takes the whole operating system down with it.

I wanted the opposite: write C on my Mac, press one button, and have the program built, copied onto the Amiga, started, and streaming its logs back to me. I also wanted an AI assistant to be able to do all of that on its own, so I could say "this game crashes after a few seconds, find out why" and watch it work. That is what Amiga DevBench does. I started it on 8 March 2026; it is now about 175 commits, runs against classic 68k AmigaOS and AmigaOS 4.1 on PowerPC, and exposes 129 tools to Claude Code through MCP.

Figure 1. The DevBench web UI’s dashboard: connection, heartbeat, free chip and fast memory, connected clients, emulator controls, and a raw protocol console

The shape of it

There are two halves, joined by a byte stream.

On the Amiga side sits amiga-bridge, a small C daemon. It opens serial.device (or, on a machine with a TCP/IP stack, a socket) and owns it exclusively. It creates a public Exec message port called AMIGABRIDGE, and then sits in a Wait() loop on three signals: serial I/O, IPC messages from client programs, and its own status window. Programs that want to take part link a tiny static library, libbridge.a, and call ab_init("my_app"). From then on they can log with AB_I("frame %ld", (long)n), expose variables with ab_register_var, publish named memory regions, and register hooks: functions the host can call into the running program. Client programs never touch the serial port themselves; everything goes through PutMsg/GetMsg to the daemon, which routes it to the host.

On the Mac sits amiga-devbench, a Python server (Uvicorn and Starlette, with FastMCP for the MCP side). It speaks the bridge protocol over a pseudo-terminal that FS-UAE opens as its serial port, or over TCP to a real Amiga or an emulator on the network, and it presents everything three ways: a web UI with a live event stream, a REST API with more than 80 endpoints, and the MCP server Claude Code talks to.

The protocol between them is deliberately boring: one line per message, pipe-separated fields, at most 1024 characters.

PING                          → PONG|2|1649344|6459000
GETVAR|bouncing_ball|speed    → CVAR|bouncing_ball|speed|i32|10
CLOG|bouncing_ball|I|1567|Frame 100

Text over a serial line is slow by modern standards, but it is debuggable with a terminal, trivially portable between 68k and PowerPC, and it never needed changing when I added AmigaOS 4 support later.

A session with Claude

The loop the tools are designed around is: write, build, deploy, run, observe, fix. Each step is a tool. amiga_build cross-compiles inside a persistent Docker container (rebuilds take a second or two), amiga_deploy copies the binary to the Amiga and checks it with a CRC32, amiga_launch starts it, and amiga_watch_logs streams its output. For the common case, amiga_build_deploy_run does all of it in one call.

Here is the kind of exchange that made me believe the whole thing was worth it:

Me:  My bouncing_ball app crashes after a few seconds.

amiga_build_deploy_run("bouncing_ball")
  → build OK, deploy OK, running
amiga_watch_logs(duration_ms=30000)
  → "Frame 100" ... then silence
amiga_last_crash()
  → alert 80000004 (access fault)
    PC 0040A2B8, A0=DEADBEEF
amiga_load_symbols("bouncing_ball")
amiga_lookup_symbol("0040A2B8")
  → bouncing_ball.c:142, draw_ball()

[reads the source, fixes an uninitialised
 pointer, rebuilds]

amiga_build_deploy_run("bouncing_ball")
  → running, no crash

The assistant does not need to see the screen to make progress, but it can: amiga_screenshot grabs the display, and there are tools to list screens and windows, inject keypresses and mouse clicks, read the palette and the copper list, and even run ARexx. When a program needs tuning, amiga_set_var changes a registered variable while the program runs, which is a lovely way to tweak a game’s physics without a rebuild.

Figure 2. The source-level debugger stopped at a breakpoint in draw_ball(): source, registers, call stack, and local variables

Two debuggers, because one is never enough

The bridge gives you a source-level debugger. Load the symbols, set a breakpoint on a function or a line, launch and attach, and you can step into and over code, walk the call stack, and read locals. There is also a GDB remote-serial-protocol server on port 2159, so m68k-amigaos-gdb or an IDE can attach the same way.

But a debugger that lives inside the Amiga dies with the Amiga. When the OS hangs, or before it has booted, the bridge is gone. So I also maintain a patched FS-UAE build that exposes the emulator’s debugger over HTTP. That one pauses the 68000 itself: it can set hardware-style watchpoints ("who keeps writing $DFF180?"), read Kickstart ROM and the CIA registers, single-step from the reset vector, and save and restore whole machine snapshots. It has no idea about source lines, but it sees everything. In practice I use them together: a watchpoint in the emulator finds where something goes wrong, and the bridge debugger lets me step through the C around it.

Figure 3. The Inspect tab: screenshot capture, palette and copper-list readers, sprite inspector, bitmap and IFF viewers, and a visual diff

The interesting problems

Most of the work was not the happy path. A few of the things the Amiga taught me along the way:

A 4 KB default stack. Every large buffer in the daemon is static. Allocating a 1 KB line buffer on the stack in a function that calls another function is enough to walk off the end and corrupt something unrelated.
printf lies about integers. In amiga.lib, %d reads a 16-bit word, so passing a 32-bit int shifts every following argument. The rule in the codebase (and in the instructions Claude reads) is %ld with an explicit (long) cast, always.
Reading memory is not always reading memory. Under FS-UAE, CopyMem returned zeros for some regions, so the memory inspector reads byte by byte through a volatile pointer. And the custom chip registers at $DFF000 are word-only: a byte read can corrupt them, so the inspector refuses to touch that range.
Never let the OS ask a question. When the host launches a program, the daemon checks the path with Lock() first and sets pr_WindowPtr to −1, so a missing volume produces an error message instead of a requester that waits forever for a click nobody will make.
No allocation on the hot path. The client library keeps a pool of four pre-allocated messages, and each client gets fixed slots: 32 variables, 16 hooks, and 8 memory regions.

Figure 4. The real thing: an A1200 board, from my SmugMug collection. Everything above has to respect what this hardware can and cannot do.

AmigaOS 4 on PowerPC

Eleven days after the first commit I started on AmigaOS 4.1, running on QEMU’s emulation of the Sam460ex board, and it has been the other half of the project ever since. The architecture held up: the serial protocol, the Python host, and the MCP tools did not change at all. The bridge needed a PowerPC build (with GCC 11 from the walkero Docker images), a TCP transport over bsdsocket.library, and a PowerPC port of the debugger and crash handler. The emulator needed rather more coaxing:

QEMU’s old -net nic,model=rtl8139 shorthand silently creates no network card on the Sam460ex machine. The working form is -netdev user,…,hostfwd=… plus -device rtl8139, which I finally wrote down in a reference config after re-deriving it more than once.
There is no chip RAM on a Sam460ex, so nothing can poke the custom chips; graphics go through graphics.library. There is also no 3D driver for QEMU’s display, so the OS4 ports of my games use a software Mesa (OSMesa 7.8.2) build.
Deploying means writing files straight into the raw hard-disk image with amitools while the OS has it mounted, which is genuinely dangerous: both sides cache the filesystem. The deploy script refuses if QEMU is running unless you insist.
And one bug I am still slightly in awe of: on the QEMU guest, a newlib memcpy of more than about 300 KB can leave its last 256 KB uncopied. Big buffers are now copied with a loop.

What it has produced

The best measure of a development environment is what people make with it. The companion repository, amiga_games, now holds about fifty programs written with DevBench, most of them with Claude doing the typing: arcade games, demos, a HAM raster test, an ARexx test, a disk benchmark, and ports of three of my 3DO games. A few of them were put together in an afternoon.

Figure 5. Frank the Frog, one of the example games built with DevBench

Figure 6. The title screen of Dot Chase, a maze game from the examples repository

There are smaller pieces too. The host can proxy an Amiga terminal program to a language model, streaming tokens back into a window on the Amiga, which means you can chat with an LLM from Workbench. And the whole environment is now three commands to set up: scripts/setup.sh, scripts/doctor.sh (which checks every prerequisite and tells you how to fix each one), and scripts/start.sh. The commercial parts, the Kickstart ROM and the AmigaOS installs, still have to come from you.

What’s next

The backlog in the repository has two big themes. The first is more examples: a list of clean-room games inspired by 8- and 16-bit classics (a Jetpac-style collect-and-launch shooter, an isometric room adventure in the Knight Lore mould, a Sentinel-like solid-3D landscape, a Lemmings-style puzzler, a Nebulus tower climber), each starting with a research phase on gameplay, graphics, and sound before any code lands, and each reproducing mechanics rather than anyone’s assets. The second is reach: the same bridge-and-MCP approach for the Atari 8-bit, with an emulator the tools can drive. The point of all of it is the line I put in the README: to make the Amiga community productive with AI. Forty years on, the machine deserves a modern toolchain.

If you have an Amiga, real or emulated, the repository has the installer, and the project page collects everything I write about it.

Transparent file encryption in the Linux kernel

2026-10-10T00:00:00Z

Most of the encryption I deal with at work happens a long way from the disk: TLS on the wire, envelope encryption in an application, a cloud provider’s "encryption at rest" checkbox. Products like Thales CipherTrust Transparent Encryption take a different position. They sit in the operating system, encrypt files as they are written, and decide per process who gets to see plaintext. The database, the search engine, the log shipper — none of them change a line of code.

I wanted to understand how that actually works, so I built a small version of it: CryptoFS, a Linux kernel module plus a Rust key daemon and admin CLI. This is the tour.

The idea: a filesystem on top of a filesystem

CryptoFS is a stacked filesystem, in the same family as eCryptfs and the classic wrapfs teaching example. It registers a filesystem type, cryptofs, and mounts one directory over another:

sudo mount -t cryptofs /data/encrypted /mnt/decrypted

The lower directory holds what actually lands on disk. The upper mount is the view applications use. Every VFS operation on the upper side — lookup, create, read, write, rename, mmap — arrives at the module first, which does its work and then calls the corresponding operation on the lower filesystem. Because it stacks on an existing directory, it works over ext4, xfs, or btrfs without reformatting anything.

Figure 1. One write() through CryptoFS: the policy check decides whether data is encrypted on the way down, or the write is refused.

Per-process policy, enforced in the kernel

The interesting part of transparent encryption is not the cipher; it is the decision about who gets plaintext. CryptoFS answers that in policy.c, on every read and write, by looking at the calling task. Rules can match on:

UID or GID;
the executable’s path, resolved from the task’s mm→exe_file;
the executable’s SHA-256 hash, computed in the kernel by reading the binary — so swapping /usr/bin/myapp for something else stops it decrypting;
the process name (comm) — convenient, and weak, because a process can set its own name.

An authorised reader gets plaintext. An unauthorised reader gets the raw extents — real ciphertext, header stripped — so a backup agent can be left unauthorised and still read the bytes without ever seeing the clear text (restoring them is another matter, as I explain below). An unauthorised writer gets -EACCES. With no policies configured at all, the module falls back to an allow-all convenience mode, which is what the demos use.

Policies and keys travel from user space over a Generic Netlink family, and the module keeps an audit trail in a kernel ring buffer.

Extents, tags, and an on-disk format

Files are processed in 4 KB extents, the same size as a page. Each extent is encrypted with AES-256-GCM through the kernel Crypto API (gcm(aes)), and its 16-byte authentication tag is stored right after it. A file on the lower filesystem looks like this:

a 128-byte header: the magic CRYPTOFS, a version, the file’s wrapped key and wrapping nonce, the plaintext length, and the ID of the master key that wrapped it;
then extent after extent of 4096 bytes of ciphertext plus a 16-byte tag.

Mapping a logical offset to the lower file is arithmetic, not a lookup: 128 + (L / 4096) × 4112 + (L % 4096). Random reads decrypt only the extents they touch, and a partial write reads the existing extent, decrypts it, patches it, and encrypts it again. The overhead is 128 bytes per file plus 16 bytes per extent — about 0.4%.

Memory-mapped files go through the page cache: read_folio decrypts one extent into one page, and writeback encrypts it on the way out. Because a page and an extent are both 4 KB, the mapping is one-to-one, which kept that code short.

Every extent’s GCM tag is checked on read, so if anything on the disk is altered — a flipped bit, a spliced block — the read fails loudly instead of returning corrupted data.

Keys: three layers

Figure 2. The key hierarchy: a passphrase unlocks the master key; the master key wraps a random key per file; each file key encrypts its extents and derives their nonces.

Master keys live in user space with cryptofs-keyd, a Rust daemon. On disk each one is encrypted under a key derived from a passphrase with Argon2id. "Unlock" decrypts it into the daemon’s memory; "activate" pushes it into the module’s key table, which holds up to 64 keys by 16-byte ID.
File encryption keys are 256 random bits from get_random_bytes, generated when a file is created and stored in its header wrapped by a master key with AES-256-GCM. Losing one file key exposes one file.
Extent nonces are not stored at all. Each is derived as HMAC-SHA256(FEK, inode ‖ extent index) truncated to 12 bytes.

There are two ways for the kernel to find a master key. In transparent mode the admin activates keys and every authorised process uses them. In guarded mode a policy binds a user to a key that is never loaded globally: the user runs cryptofs-admin key session-unlock, which puts the key into their session keyring, and at open time the module calls request_key() for cryptofs:<key-id>. If the key is not in that process’s keyring, the open fails with ENOKEY. It is the closest thing in the project to "only Alice can read Alice’s files, even though root mounted the volume" — with the usual caveat that root on a live system can still reach almost anything.

The daemon talks to keys through a KeyProvider trait. Only a local provider exists today; the trait is there so an external KMS — HashiCorp Vault’s transit engine, AWS KMS, or my own SecureVault — can be slotted in later. That integration does not exist yet.

What I would change: the nonce

Writing this up made me look hard at the one decision I am least comfortable with.

Deriving nonces from the file key, inode number, and extent index means nothing needs storing and two extents never share a nonce. But an extent that is rewritten keeps its nonce. If it is rewritten with different data, AES-GCM is used twice with the same key and nonce — the cardinal sin of GCM. An attacker who captures the lower file before and after the rewrite learns the XOR of the two plaintexts for that extent, and nonce reuse also weakens GCM’s authentication. The README’s note that "the same data produces the same ciphertext" is true, but it understates the problem: the risk is different data under the same nonce.

The inode number in the derivation has a second consequence: the ciphertext is bound to the lower file’s inode. Copying the lower files to another filesystem — a backup restored with cp or rsync — gives them new inode numbers, and their extents no longer decrypt.

The fix for both is well understood: store a random nonce per extent (12 more bytes per 4 KB), or keep a write counter per extent and mix it in, or move to a construction designed for this — an XTS-style scheme for confidentiality plus separate integrity, or AES-GCM-SIV, which tolerates nonce reuse. That is the next change I would make, and it is the kind of flaw that only shows up when you are forced to explain your design to someone else.

How it is tested

Kernel code is unforgiving, so most of the repository is tests: shell suites for basic file operations, random access and extent boundaries, appends, concurrency, UID policies, guarded keys, and Docker containers (an authorised UID 1000 container sees plaintext; an unauthorised UID 65534 one sees ciphertext). A stress test runs graduated file sizes up to 500 MB, patterned and random data, overwrites, random-offset patches, concurrent workers, file churn, and unmount/remount persistence, checking everything with SHA-256. fio benchmarks compare the encrypted mount with a plain baseline, and build_and_test.sh runs the whole pipeline end to end.

There is also a demo of Apache Solr 9 running in Docker with its index on a CryptoFS mount — no changes to Solr at all, which is the whole point of doing this in the kernel.

Trying it — in a VM

Do not load an experimental kernel module on a machine you care about. On a Mac, the repository includes a Vagrant definition for an ARM64 Ubuntu 24.04 VM under QEMU with every dependency installed:

brew install vagrant qemu && vagrant plugin install vagrant-qemu
make vm-up && make vm-ssh

# inside the VM
make all
sudo insmod kernel/cryptofs.ko
sudo daemon/target/release/cryptofs-keyd --foreground &
cryptofs-admin key generate --label demo     # then key unlock / key activate <id>
mkdir -p /tmp/lower /tmp/mnt && sudo mount -t cryptofs /tmp/lower /tmp/mnt
echo "secret data" > /tmp/mnt/hello.txt
cat /tmp/mnt/hello.txt      # secret data
xxd /tmp/lower/hello.txt    # CRYPTOFS header, then ciphertext

It needs Linux 6.1 or later with CONFIG_CRYPTO_AES and CONFIG_CRYPTO_GCM.

An honest security note

CryptoFS is a learning project and a working prototype. It has not been audited, the nonce scheme above is a real weakness, and it protects data at rest only: it does nothing against a compromised kernel or root on a running system with keys loaded. If you need this in production, use dm-crypt or LUKS for whole volumes, fscrypt for per-directory encryption built into ext4 and f2fs, or a commercial transparent-encryption product.

What it did give me is a working model of how those products are built: a stacked VFS layer, a policy decision made from the calling task, a key hierarchy that keeps master keys out of the kernel until they are needed, and an on-disk format that can be read at any offset. The source is on GitHub.

Writing 3DO games in 2026: a 1993 console, a 1990s compiler, and a debuggable emulator

2026-10-10T00:00:00Z

The 3DO Interactive Multiplayer came out in 1993 and asked for $699. It had an ARM CPU when everyone else was on 68000s and MIPS, a sprite engine that thought in scaled and rotated quadrilaterals, and a real preemptive multitasking OS underneath every game. It is a lovely machine to program, and in 2026 it is surprisingly hard to get started on.

So I built 3do-dev: a dev kit that turns "write a 3DO program" into a modern loop of edit → build → boot → observe → test → debug, which works the same whether the thing at the keyboard is me, a script, or Claude Code. This is how it works and what it took.

Figure 1. The demo: plasma, starfield, a rotating cel vortex, a bouncing title, and a sine scroller, at 60 fps on an emulated 12.5 MHz ARM60.

Why the 3DO is an interesting target

The hardware in one paragraph: an ARM60 at 12.5 MHz (big-endian ARMv3), 2 MB of DRAM plus 1 MB of VRAM, and two custom chips. MADAM drives the cel engine, which draws hardware-scaled, rotated, and blended sprites ("cels") and does matrix maths; CLIO handles video, the audio DSP, timers, and I/O. Games don’t bang the hardware directly. They run on Portfolio, a preemptive multitasking OS with tasks, message ports, semaphores, and shared libraries called folios (graphics, audio, maths, filesystem). Programs are AIF executables on a CD whose boot code is RSA-signed.

That combination — a RISC CPU, a quad-drawing GPU ancestor, and a proper OS — makes it feel much closer to modern programming than its contemporaries, while still being small enough to understand completely.

Why it is hard in 2026

Three things stand in the way.

The compiler. The only complete toolchain is ARM’s own SDT 2.51 — the Norcroft C compiler that 3DO developers actually used. trapexit’s 3do-devkit packages it with the Portfolio 2.5 SDK and modern disc tools, which is a huge head start. But the compilers are 32-bit i386 Linux binaries and the disc tools are x86-64 Linux binaries. Nothing runs them natively on a Mac, and Rosetta doesn’t do 32-bit.

The feedback loop. Emulators exist — Opera, the maintained descendant of FreeDO and 4DO, is excellent — but an emulator built for playing games gives you a window and nothing else. No debug console, no breakpoints, no way for a script to press a button and check what happened.

The debug information. Norcroft emits DWARF 2, but in ways modern gdb quietly rejects.

How the kit solves it

A 1990s compiler in a container

bin/3do-make runs make inside a tiny amd64 Debian image with the i386 C library, with the repo mounted and running as my own user ID, so generated files belong to me. On Apple Silicon the build script registers QEMU user-mode emulation for i386 and x86-64 with the kernel’s binfmt_misc, so the container’s x86 binaries just run. A build still takes about two seconds.

The shared rules in sdk/project.mk compile with armcc -g -O2 -bigend -apcs 3/32/nofp/swst, link a self-relocating AIF with armlink, stamp the 3DO header with modbin, copy in the base filesystem, and run 3dt pack to build an ISO and RSA-sign it, so the retail BIOS boots it. Then the same objects are linked a second time as an ELF with DWARF at address 0x80, so ELF addresses equal offsets into the loaded AIF. That second link is what makes symbols and gdb possible.

An emulator you can drive

The emulator is libretro Opera, compiled natively with a patch of roughly 50 lines that adds hooks for the debug console, breakpoints, watchpoints, single-step, system-call tracing, and memory access. The hooks only cost anything while they are armed.

The part I’m proudest of: stopping the CPU never disturbs emulated time. Opera emulates a frame by running the CPU in slices interleaved with scanline processing. I made that frame function resumable — when the debugger says stop, it saves its scanline and cycle position and returns early, and the next call continues the same frame exactly where it left off. From the 3DO’s point of view, no time passes while you sit at a breakpoint.

A small Python libretro frontend (ctypes, no C glue) loads the core in-process. All the heavy work stays in C; Python only runs between frames, so headless emulation still reaches about 900 frames a second — fifteen times real time. Each emulator lives in its own session process with a JSON control port, a GDB remote-protocol server, an optional window, and a registry entry so other tools can find it.

Seeing inside Portfolio

Because Portfolio loads your program wherever it finds memory, the kit finds it by searching emulated DRAM for the program’s code (AIF self-relocation replaces a BL in the header with a NOP, which confirms the match). Runtime address = load base + ELF address, and suddenly break update_plasma, read_u32 addr=s_palette, and gdb’s qOffsets all just work.

It goes further and reads the OS’s own data structures. Every Portfolio object is an item; walking the kernel’s item table gives tasks, message ports, semaphores, devices, I/O requests, screens, and a memory map of every DRAM and VRAM page by owner. The structure offsets come from a tiny 3DO program that prints offsetof() for the SDK structs, so they are right by construction. None of this needs any code on the 3DO, and it works on a crashed program.

Figure 2. DevBench’s OS inspector: the Portfolio task list, read straight out of emulated RAM.

All of that is exposed three ways: the ./3do ctl command line (about 70 commands), a browser DevBench with a REST API, and an MCP server with about 50 tools so Claude Code can build, boot, press buttons, look at the screen, read logs, set breakpoints, and run gdb on its own.

Figure 3. Debugging Nova Defense in DevBench: stopped at a breakpoint, with the live system-call trace and profiler.

A walkthrough

Installing is one line — it checks prerequisites, clones the repo, fetches the toolchain, builds the emulator, runs the test suite, and opens the demo:

curl -fsSL https://raw.githubusercontent.com/geekychris/3do-dev/main/scripts/install.sh | bash

From there, a new game is a template away:

./3do new mygame               # copy the template into projects/mygame
$EDITOR projects/mygame/src/main.c
./3do run mygame               # rebuild, boot, open a window (control :7330, gdb :2330)
./3do test mygame              # headless smoke test
./3do gdb mygame               # source-level debugging

While it runs, everything is scriptable over the control port:

./3do ctl press buttons=A
./3do ctl screenshot path=shot.png
./3do ctl break addr=update_plasma
./3do ctl watch addr=s_palette     # stop when a global changes
./3do ctl save_state name=before-boss

And gdb behaves like gdb:

(gdb) break main.c:396
(gdb) continue
Breakpoint 1, update_text (t_=1284) at src/main.c:396
(gdb) p *s_squares[0]
$1 = {ccb_Flags = 1063666704, ccb_NextPtr = 0x84670, ccb_SourcePtr = 0x88a44, ...}

The programs themselves are C89 for Norcroft: no FPU (16.16 fixed point and the operamath folio), big-endian, and graphics as a linked list of cels drawn with one DrawCels() call per frame. The demo draws about 110 cels per frame at 60 fps alongside a 64×48 per-pixel software plasma.

The hardest problems

Making gdb believe Norcroft. The DWARF was valid-ish but rejected in three silent ways, which tdo.elffix repairs after linking:

armlink names every loadable section after the output file, so gdb relocated code but not data — and globals were read from, and written to, the wrong addresses. Fix: rename sections by their flags to .text, .data, .bss.
DW_AT_stmt_list used DW_FORM_addr instead of an offset form, so gdb ignored all line-number information. Fix: patch the abbreviation to DW_FORM_data4 — same size, so nothing else moves.
The .debug_frame call-frame info used conventions gdb misreads, giving one-frame backtraces. Fix: hide it; gdb’s ARM prologue analyser unwinds this code correctly on its own.

Debug output that lies. The kernel writes debug text a byte at a time to a MADAM register, which the harness captures. But kprintf() only formats its first three arguments correctly: it’s a SWI, so arguments four and up land on the user stack, where the kernel never looks. I spent a while doubting my own arithmetic before working that out. The kit’s tdo_log() formats with vsprintf and passes a single string.

Cel engine details. Clear CCB_LAST on every cel except the last or the engine stops early; HDX/HDY are 12.20 fixed point while the rest are 16.16; a pixel value of 0 is transparent, so opaque black has to be 0x0001. All of these are now written down in the repo’s notes, verified in the emulator.

Results

The demo was written, built, tested, and debugged entirely through the kit. Then it got ambitious: an Amiga compatibility layer — a Paula mixer streaming to the 3DO audio folio, a C ProTracker player, and fast paths for RectFill, polygon fills, and run-length sprites — let me port eighteen Amiga games, and three Unity games were rebuilt for the cel engine: Planet Chomp, Spectral Keep, and Rolling Steel. Each has its own disc, and all of them sit on an arcade disc behind a menu.

Figure 4. The arcade disc: every game behind one menu.

Figure 5. Spectral Keep — an isometric Unity adventure, rebuilt for the 3DO’s cel engine.

Figure 6. Planet Chomp, the Unity sphere-maze game on the 3DO.

Figure 7. Canyon Rescue, ported through the Amiga layer.

The repository’s games gallery measures each one: many run their game logic at the Amiga’s 50 steps a second, with drawing rates from about 9 fps for the most demanding to around 49 for the simplest.

Next steps

The kit is new — the first commits landed this month — and the obvious next steps are about speed and polish: more fast paths in the Amiga layer for the slower ports, and more of the hardware notes turned into tests. The bigger lesson is the one I keep relearning on retro platforms: the machine is rarely the hard part. The hard part is the loop around it, and once that loop is fast and scriptable, both people and agents get a lot braver.

The code, docs, and every game are at github.com/geekychris/3do-dev.

First build of the new site

2026-10-09T00:00:00Z

The first version builds every page from AsciiDoc in a single Asciidoctor process, caches the rendered HTML by content hash, and publishes by copying only the files it owns into the GitHub Pages repository — so the books and other projects that already live there are left alone.

Building a full social network from scratch: feed, friends, groups, DMs, search

2026-05-12T00:00:00Z

Most modern social platforms aren’t a single thing — they’re a stack of intersecting systems (feed ranking, social graph, search, messaging, moderation, analytics) that have grown together over years. I wanted to know what it actually takes to build all of that as one coherent app, end-to-end, with no shortcuts. The result is the Social Enterprise Platform: a full-featured social network built with Spring Boot, React, PostgreSQL, OpenSearch, and a custom Rust social-graph cache.

What’s in it

Far more than I expected when I started. The app ships with:

Personalized feed with 80% organic + 20% recommended content, six reaction types, nested comments, link previews, cursor-based infinite scroll
User profiles with follow/unfollow, follower lists, activity history
Friend requests with mutual-follow semantics
Groups — public or restricted, role-based (Owner / Admin / Member), pending approval flow, pinned posts
Pages for organizations or topics, with their own feeds
Direct messages with real-time conversations and file attachments
Search across users, groups, and pages via OpenSearch (with PostgreSQL fallback)
Admin dashboard with DAU/WAU/MAU, engagement analytics, moderation tools
AI support with Lucene KNN vectors, answer caching, knowledge-gap analysis, and feedback loops

The interesting parts

Postgres handles the durable social graph, but reading “who follows whom” on every feed request is too slow at any reasonable scale. AOEE is a small Rust service that holds the graph in memory with efficient bitset operations and answers reachability + mutual-friend queries in microseconds. Updates flow in via Postgres LISTEN/NOTIFY and a write-behind queue.

Feed ranking as a pluggable mix

The 80/20 organic-to-recommended split isn’t hardcoded; it’s a ranking pipeline with explicit stages (candidate generation, scoring, diversification, blending). Each stage is testable in isolation. The recommended path uses Lucene KNN vectors over a content-embedding index.

AI support that improves over time

The latest iteration adds a full AI support system: an intent classifier routes user questions, a retrieval layer pulls relevant context from the platform’s knowledge base, an LLM answers, and every interaction feeds back into a knowledge-gap dashboard. When too many users hit the same gap, an admin sees it surface in the metrics view.

What I learned

Social-app surfaces are deceptively simple. A “comment” looks like a row in a table; in practice it’s three database rows, a search-index update, an OpenSearch refresh, a webhook fan-out, a moderation check, and a feed-cache invalidation. Building all of that yourself is humbling. But you understand the system in a way no managed SaaS ever lets you.

Source

github.com/geekychris/enterprise_social_platform — 34 commits in, with full screenshots, architecture docs, and a Docker Compose stack that brings up the entire system locally.

Architecture diagrams

Rendered from the Mermaid blocks in the project README so the post stays in sync with the source of truth.

ViewScene: a desktop Luke alternative for inspecting Lucene indexes

2026-05-11T00:00:00Z

If you’ve spent any time with Apache Lucene as a developer, you’ve probably wanted to peek inside an index. Maybe a query isn’t matching what you expected. Maybe the analyzer is doing something surprising. Maybe a vector field has the wrong dimensionality. The classic answer is Luke, the Lucene Inspector — but Luke is a web app these days, and I missed having a real desktop tool.

So I built ViewScene: a standalone JavaFX desktop application for inspecting and diagnosing Lucene indexes. Self-contained fat JAR. Native desktop UI. No web server, no browser, no external dependencies beyond a JDK.

What it shows you

Fields browser — every field across the index, with type, doc count, indexed/stored flags, vector dimensions if any
Document viewer — pick any doc by ID, see every field’s stored value, indexed terms, and term positions
Term dictionary — the actual postings list per field, browsable and searchable, with document frequencies
Search — run real Lucene queries against the open index, see results ranked with explanations
Vector search — for KNN-indexed fields, run a vector query (with explicit embedding input or text-to-embedding via a configured analyzer)
Analysis pipeline — type in text, pick a field, see exactly what tokens come out at every analyzer stage. The most useful tab when you’re trying to understand “why doesn’t this match?”
Index tools — segment listings, commit history, optimize / force-merge / delete operations (read-write mode is opt-in)

The interesting parts

Fat JAR with no install

Download one file, double-click, you’re inspecting an index. No Maven repo to bootstrap, no Node modules, no server to run alongside. The whole thing — UI, Lucene core, vector support — gets packed into a single executable JAR via the Maven Shade plugin with merged service files. Total weight is around 50 MB, which is mostly the Lucene + JavaFX runtime.

Direct API access, not a query layer

Many index inspectors are wrappers around a query DSL. ViewScene uses the Lucene API directly: IndexReader.leaves(), Terms.iterator(), PostingsEnum, FloatVectorValues. The UI panels are thin views over those APIs. That keeps the tool honest — if Lucene shows it, ViewScene can show it. If Lucene doesn’t, neither does ViewScene.

Vector search with explainability

The vector-search panel does what Luke historically hasn’t: lets you submit a query vector (either as numbers or as text routed through an analyzer + embedding hook), runs a real knnFloatVectorQuery, and shows the actual distance scores per hit. Combined with the analysis pipeline tab, this turned out to be invaluable for debugging KNN-based “AI search” features: you can see why doc 47 ranked above doc 12 in actual numbers.

Origin: hitorro

The core viewer module was originally part of the Hitorro project. I extracted it into a standalone, fully independent tool so it could be used outside that context. The base was clean; the standalone work was mostly stripping coupling and packaging.

Why this matters

If you build anything on Lucene, you eventually need to look inside an index. ViewScene exists so that “look inside an index” is one double-click away — same as opening a database GUI. The web-based alternatives are great when you have infrastructure for them; for the case where you just want to drag-drop an index path and look at fields, a desktop tool wins.

Source

github.com/geekychris/lucene_viewer. JavaFX, fat JAR, MIT license.

Building SolarSage: a local-first solar monitoring tool

2026-05-10T00:00:00Z

I have an off-grid in San Felipe, Mexico, with a 15 kW EG4 solar system and a wall of lithium batteries. When the grid isn’t there, “do I have enough power to run the AC tonight?” stops being abstract. The vendor dashboard at monitor.eg4electronics.com shows live numbers and a few canned charts, but it doesn’t predict anything. I wanted a tool that did.

So I built SolarSage — a local monitor that polls my system every minute, captures everything in SQLite, layers a weather-aware forecast on top, and turns “you’ll have surplus solar” into “run the water heater between 12 and 3 PM tomorrow.” It runs entirely on my laptop, with no cloud component and nothing leaving my network.

The bigger goal is intelligence: predicting how much energy I’ll produce, how much the AC will draw at any given outdoor temperature, and where the headroom is. Off-grid living rewards that kind of forecasting in a way grid-tied living doesn’t — if you guess wrong, you run out, or you run a generator.

What it does

The architecture is intentionally boring: a FastAPI backend polls the EG4 portal every 60 seconds and writes every numeric field to a SQLite database. A small React UI reads from the backend. Forecasts and analytics layer on top. Everything runs on 127.0.0.1; no cloud, no telemetry, no signup. Credentials live in a 0600 file on my laptop and that’s it.

The interesting parts are the things that surprised me on the way.

1. EG4 has no public API — but it has chart endpoints

The official “API documentation” page is marketing. What the portal actually has is a set of undocumented JSON endpoints that power the web UI. After some reverse-engineering (and reading other people’s work), I found that /WManage/api/analyze/chart/dayLine with attr=ppv1 returns full-day timeseries for any attribute — but only some attributes work, and they vary by firmware. On my SNA-US 15K, ppv as a chart attr returns empty; I have to sum ppv1+ppv2+ppv3 myself. The “all channels at once” endpoint other libraries use (dayMultiLineParallel) returns data: [] entirely on this firmware.

The real gotcha: the chart endpoint’s month field is Java zero-indexed. May = 4. I caught it because timestamps came out exactly one month off — a great way to lose an afternoon if you trust the field names.

2. Free weather is finally great for solar

Open-Meteo is the only free, keyless weather service I’ve found that returns forecasted shortwave irradiance (GHI), not just cloud cover. That changes the modeling. Instead of estimating PV from “30% clouds”, I multiply the forecast GHI by my system’s measured W-per-W/m² calibration (about 9.08 for my install) and get tomorrow’s predicted PV curve hour by hour. The historical archive endpoint lets me back-fit other models too.

3. The AC model is one line of math

My biggest variable load is air conditioning, and AC tracks outdoor temperature almost linearly above the comfort threshold. SolarSage fits a cooling-degree model from joint Open-Meteo + load history:

load(hour) = base(hour) + slope_W_per_°F · max(0, °F − threshold)

Grid-search the threshold from 50°F to 90°F, OLS-fit the slope on the residual. After 15 days of data my model lands at threshold=71°F, slope=78 W/°F, R²=0.43. Not yet impressive — but better than the constant-load assumption every other tool uses, and it sharpens automatically as more days accumulate.

4. The dashboard already caught something

The string-health panel watches each PV string’s daily ratio against the strongest. Within hours of finishing it, the panel flagged ppv3 = 0% across every day. Either I have an unused third string (likely), or a fuse blew, or there’s a wiring fault. Either way, the alert came from the system itself — exactly the kind of slow drift the EG4 portal hides in plain sight.

5. Forecasts → concrete recommendations

The “smart load scheduler” turns the surplus forecast into actionable windows: for each enabled appliance, scan the next 48 hours for a slot where every hour clears the appliance’s watts — not just the average. Right now it’s telling me to run the water heater + EV charger + dishwasher all in the same 12-15:00 block tomorrow, because that’s where 7.7 kW of sustained surplus lives. Pool pump is disabled because I don’t have one; electric dryer is disabled because mine’s gas. The scheduler respects what I told it I actually have.

Try it

Source is at github.com/geekychris/solarsage. The one-liner install is:

curl -fsSL https://raw.githubusercontent.com/geekychris/solarsage/main/install.sh | bash

It runs on macOS, Linux, and Windows. The architecture doc has Mermaid diagrams of the data flow; the installation doc has manual steps if you’d rather not curl | bash. The Q.Cells adapter is stubbed pending information on which monitoring portal Q.Cells installs actually talk to — drop me a note if you have one.

Building the Amiga 40th: a carousel of progress

2025-08-01T00:00:00Z

The Amiga turned 40, and we wanted to do something special about it.

Bill Borsari and I had been talking informally about doing something for a while, but the real start was the last day of AmiWest 2024. On the way back from the show, Bill and I had a multi-hour conversation and started to flesh out the idea. When I got home I wrote up a "treatment" of the show, and over the next few weeks I sketched out my thoughts while Bill started sketching the floor layout.

We talked to Zach Hardesty from the Vintage Computer Festival, and he was fully supportive of helping us celebrate the 40th at the Vintage Computer Festival at the Computer History Museum. Over the following months we talked with the VCF staff, who at first had a hard time understanding what we were trying to do. Perhaps that’s because, up to then, most VCF exhibits had been a few treasured machines on a table. A fully immersive exhibit, more like a pop-up museum, took a little more imagination. :-)

A carousel of progress, for the Amiga

We wanted to tell the story of the Amiga, and I proposed modelling it on Disney’s Carousel of Progress: walk through a series of scenes, each one a moment in time.

It starts at the first CES that Amiga attended. Out front, Amiga showed off joysticks and games for 8-bit computers. Behind a curtain, by invitation only, was the Amiga computer itself, still in its TTL wire-wrap prototype form. So our exhibit began with a facade of the CES 1984 booth, and a curtain took you through to the wire-wraps.

The CES exhibit, complete with video games.

The original Amiga TTL wire-wrap prototype boards.

Some of the black devbox tooling.

Part of the prototype demo.

The bedroom and the NewTek video scenes, side by side.

Alongside them was the tooling it took to run the Amiga, including a Sage computer, and some of the black devbox work, in operation.

From there you walked into a reenactment of a typical bedroom, where so many kids had an Amiga 500 to play games and "help do homework". Next door we paid homage to the Video Toaster with a recreation of a video booth and office.

The NewTek booth.

Then the Future Wall: lots of new Amigas, all stripped bare and mounted on a single wall, and synchronised so they worked as one big display. We also showed the industrial design work of Rob Gemmell. Each scene had a kiosk playing a video that explained it. Bill produced the videos, with some help from AI; the scripts and ideas were all his. I built the kiosks and some of the software that drove the videos.

Building it at home

Almost all of the exhibit was made at my home, with help from Winnie, my son Kyle and Alan Swithenbank. The planning started after AmiWest in October, but nearly all of the real making didn’t begin until July.

The hardest part was the trial and error of working out how to put it all together quickly and on budget. I did a lot of prototyping for things like the CES facade and how we would construct the bedroom. The art for the large prints in the bedroom and the video room was done by Evangelos Goulas.

Alan building the frame that housed the demo monitor in the secret prototype room.

Part of the CES facade, loaded onto Dale’s truck.

The Amies trophies

In the evening we held the Amies, giving out awards to people who have contributed to the Amiga. There were lots of small awards for different kinds of contribution, some very small ones to give away, and one very large one that served as a display piece and for people to hold in photographs.

The trophy is a lemming. It was designed in LightWave by an Amiga fan in eastern Europe, because at the time we couldn’t get AI to produce a model that was printable. I printed, cleaned and painted them.

A giveaway lemming beside the full-size one we used for award-ceremony photos and promotion.

The finished full-size lemming.

Getting the boing-ball detail right.

Winnie with a tray of giveaway lemmings, after a lot of post-print cleanup and gluing.

Each award carried a chip that said what it was for, so of course we made a giant one too.

Winnie putting the finishing touches on our massive chip.

The super-sized chip. Smaller ones on each award said what it was for; this one went with Bill’s lemming outfit to promote the evening.

The Amies.

Mama Lemming.

The Future Wall

The Future Wall needed lots of Amigas to behave as one display. Chris Hopper and his son designed a device that plugged into every Amiga through its joystick port and could strobe through each one in turn. An ARexx script on each Amiga reacted to the trigger and changed what it showed.

Laying out the Future Wall.

The Future Wall.

Moving in

The empty floor, just before we turned it into CES 1984.

Michael Slifak and Trevor Dickinson helping to set up the exhibit.

Michael, Don and Dale putting up the official Amiga sign.

Michael, Hedley and Jeff adding the finishing touches.

Tim Jenison, founder of NewTek, signing our visitor banner.

Jeff Bruette signing the banner. Jeff worked at Amiga and taught Andy Warhol to use the Amiga for the launch.

Colin Proudfoot adds his name.

We also put up a banner for visitors to sign, and some of the first signatures came from people who made the Amiga’s history.

We did drop a few things at the last minute. We had planned more parts to the exhibit, but time and space ran out.

The day

The CES 1984 facade: the entrance to the Amiga 40th.

People were amazed. Some 2,500 came through.

Some 2,500 people came through.

Playing at the CES exhibit with an Amiga joypad.

Amy Herndon enjoying our fluffy red carpet. It was a bit like dancing on a polar bear.

Your browser does not support the video tag.

Jeri Ellsworth living out her childhood in our bedroom scene.

An Amiga-based arcade machine from Dale Luck’s collection.

I showed Leonard Tramiel and Joe Decuir around personally. They both responded like kids at Christmas. Their eyes lit up, and they shared story after story.

With Joe Decuir, Amiga’s original systems architect (Dave Needle later took over the role), talking about the early days. He was like a kid in a candy shop.

With Rob Gemmell, who did the industrial design for the Apple IIc and went on to draw the first designs of what became the Amiga 2000 (Ranger, which was about packaging, not a new chipset!) and the Amiga 500.

With Paul Lassa, who designed the Amiga 4000T and the C65.

Tim Jenison and Bill repairing a NewTek Video Toaster.

One visitor got so excited that he started "panic explaining" to his girlfriend what the Amiga meant to him. He mentioned playing on his 500 in his bedroom. At that point I said "hold on" and walked him over to the bedroom scene, where he actually started crying.

The special thing for me was all the original Amiga and Commodore employees who helped. It was also the last time I got to see Hedley Davis, who sadly passed away not long after. Hedley spent most of the day explaining the TTL hardware to visitors.

The evening

Bill drumming up business for the evening event.

Your browser does not support the video tag.

In the evening we paid homage to the 1985 Amiga launch, and held the Amies.

The evening: our homage to the 1985 Amiga launch, and the Amies award ceremony.

Amiga Bill, Bill Borsari and Trevor Dickinson.

Tim Jenison and Kiki Stockhammer of NewTek at the Amies, with Kiki playing the role of Debbie Harry.

Not everything went to plan. For the 1985 launch homage we were going to use Amiga speech to talk back, but we had technical issues, and problems with the video setup and feed caused a long delay in the evening event.

The Amiga 40th team after the Amies, in tuxedos as a nod to the original 1985 launch at Lincoln Center.

Teardown

Ron Nicholson helping to tear down the exhibit.

It was amazing to see my personal idols taking part. We did this to celebrate them, and at the same time they all rolled up their sleeves and made the event fantastic.

Thanks

The biggest thanks goes to Bill Borsari, who was an amazing partner to work with: from the conversation that started it all to the kiosk videos. Everyone else who made it happen is on the project page.