System diagnosis and methodology

A broken system does not answer the question "what is wrong". It answers the question you ask it with a command. Half of diagnosis is asking the right question; the other half is not changing ten things at once and then guessing which one helped.

The method is the same for a hung service, a slow query, and a process eating the disk. Reproduce the problem. Narrow the area: where exactly does it break - in the network, in a file, in permissions, in the program itself. State one hypothesis. Test it by changing exactly one thing. If it does not hold, roll the change back and take the next hypothesis. The rollback is not politeness, it is a condition: otherwise five steps in you no longer know what state the system is in.

The first source of facts is the systemd journal. It gathe