Linux triage

When a Linux box (or a container) is slow, the first minute is the same every time: a handful of standard commands, read in order, looking for saturation and errors. This page does the reading. It follows Brendan Gregg’s Linux performance analysis in 60,000 milliseconds and his USE method: for every resource, utilisation, saturation and errors.

How to use it

Run the commands (the page lists them, with a copy button, and a version that labels each output) and paste everything into the box — in any order, with or without the command lines. It recognises:

uptime, dmesg (with or without -T), top -b
vmstat 1 (including -w and -t), free (old and new layouts, any unit), /proc/meminfo
mpstat -P ALL 1, pidstat 1 (and -d, -r, -w), iostat -xz 1 (sysstat 10 to 12 column layouts), sar -n DEV / EDEV / TCP,ETCP / SOCK, sar -u / -q / -r / -B / -W
/proc/pressure/{cpu,memory,io} (PSI) and the cgroup *.pressure files
ss -s, ss -tin, ss -ltn (accept queues)
cgroup v2 memory.events, memory.stat, memory.max/current/peak, cpu.stat, cpu.max (and v1 cpu.stat)
docker inspect state and kubectl describe pod termination blocks

For each output you get the findings (with the thresholds that matter, e.g. vmstat r against the CPU count, iostat await and aqu-sz rather than %util on parallel devices, retransmits as a share of segments sent, cgroup throttling per period), what every column means, and the commands to run next. On top, a USE table fills in what the paste measured for CPU, memory, disks and network.

Exit codes & signals explains any status: 126/127, 128+n, 130, 137, 139, 143 and the rest, the signal table, and how to tell an OOM kill from another SIGKILL in Docker, Kubernetes and the kernel log.

Examples are real outputs I captured for this page in throwaway containers under stress-ng (Ubuntu 24.04 and Rocky Linux 8), a container OOM-killed at a 64 MiB limit, and one throttled at half a CPU.

What I checked

The parsers are tested on outputs from Debian 12, Ubuntu 20.04 and 24.04, Rocky Linux 8 and Alpine 3.20 — procps 3.3.15 to 4.0.4 and sysstat 11.7 to 12.6, whose iostat columns differ — with the averages recomputed independently in the tests. Two observations from those runs are written into the readings: in my Docker 29 setup a container killed by the memory-cgroup OOM killer reported ExitCode=137 with OOMKilled=false (the kernel log and memory.events had the kill), and a sleep running as PID 1 exited 137 on docker stop but 143 with --init.

Related: strace viewer (which system calls a process makes), flame graphs (where the CPU goes), log parser, server config (systemd units), container dashboard, Kubernetes manifests (resource limits).

Browser support

Any current browser.

Privacy

Everything is read in this page; nothing is uploaded or stored except the last example you opened and the exit code you looked up (in this browser’s local storage).