Sep 19, 2026 | Client work, Networking, Reliability
Five WordPress sites on one shared server started returning 502s. They stayed down for about twenty hours. Nothing had been deployed, nothing had been upgraded, and the server was up the whole time. What happened The server runs each site’s PHP in containers...
Sep 18, 2026 | Infrastructure, Reliability, Self-hosting
Logs are the thing everyone agrees they should centralise and nobody wants to pay for. The usual outcome is that the noisy source drowns the useful one, retention gets cut to a week, and the one event you needed is three days past the edge of the index. I centralised...
Sep 18, 2026 | Infrastructure, Reliability
Half the drives in my fleet had no reported age, no wear level and no temperature. Not wrong values — no values. And I did not know, because the dashboard showed the drives that did report, and a missing row does not look like anything. Why the standard collector was...
Sep 18, 2026 | Infrastructure, Reliability
A replicated storage cluster’s core promise is that a copy of your data lives on more than one machine, so losing a machine loses no data. That promise is enforced by a placement rule, and the rule has a setting for what counts as a separate failure domain: a...
Sep 15, 2026 | Reliability
There is a specific kind of confidence that comes from a backup job with a green checkmark, and it is unearned. A backup is a hypothesis. The test is a restore. The databases in my cluster were being captured by filesystem-level replication — the volumes were mirrored...
Sep 11, 2026 | Infrastructure, Reliability
Infrastructure-as-code tools keep a state file describing what they believe exists. Most backends support a lock, so two runs cannot write that file at once. The object-storage backend I use does not. I knew that. I had read it. I did not understand what it meant...