There is a specific kind of confidence that comes from a backup job with a green checkmark, and
it is unearned. A backup is a hypothesis. The test is a restore.
The databases in my cluster were being captured by filesystem-level replication — the volumes
were mirrored offsite, so in a sense everything was “backed up.” That is the exact arrangement
that fails when you need it, because a copy of a live database’s files, taken while it is
running, can be torn: half a transaction, an index that does not match the table, a write-
ahead log referencing pages that were never copied. It restores into a corrupt database, and you
find out at the worst possible moment.
Every database dumps itself, with its own tools
Each application now runs a small sidecar container alongside it, built from that database’s own
image, that produces a proper logical dump on a schedule. PostgreSQL gets dumped by PostgreSQL’s
tools at PostgreSQL’s version. MariaDB likewise. Nothing tries to be clever or generic about it.
I considered a single central job that would reach into each database and dump it, and chose
per-application sidecars instead. The central version is less code and more coupling: it needs
credentials for everything, it needs every client version, and one bad upgrade breaks every
backup at once. The sidecar version has more moving parts and no shared failure.
One hard-won rule from this: never run the dump inside the database’s own container.
A database process that reaps a foreign child process it did not create can interpret it as a
crashed worker and drop into crash recovery. That is a backup job causing an outage, which is a
uniquely frustrating way to have a bad day.
Two directions offsite, and one of them cannot be deleted
Dumps land on cluster storage, then go two ways:
- To a physical machine elsewhere, which keeps snapshots on its own schedule.
The mirror is not the backup; the snapshots on the receiving end are, because a mirror faithfully
replicates a deletion. - To object storage with a thirty-day lock, using a key that has no delete
permission at all. Ransomware that owns the cluster still cannot remove those objects, and neither
can I, and that is the point.
The copy is a copy, never a sync. A sync would propagate exactly the deletion you are trying to
survive.
The restore tests
Every month, a job takes the newest dump for each database engine, stands up a throwaway server,
and restores into it. If the restore fails, that fails loudly.
This is the part most setups skip, and it is the only part that converts a hypothesis into a
fact. It is also cheap: a scheduled job, a temporary volume, and a comparison. The reason it gets
skipped is not cost. It is that a green backup job already feels like proof.
Two things that were quietly wrong before this
A backup job can succeed against the wrong target. On a client’s cloud
infrastructure I found a backup policy that had been running cleanly, on schedule, with no errors
— protecting a virtual machine that had been deleted. The live machine that replaced it had no
protection at all. Both facts had been true for months and the dashboard was green throughout,
because the dashboard was reporting on the job, not on the estate.
Local copies can fail for months in plain sight. On one hypervisor, local
backup copies had been failing since April. The remote copies were fine, so nothing ever went red
at the top level, and the per-node detail was not on anything anyone looked at.
Both of those are the same failure as the first one: monitoring the mechanism instead of
monitoring the outcome. “Did the job run?” is easy to answer and nearly worthless. “Is
everything that should have a backup, backed up, and have we put one back recently?” is the
question, and it costs more to answer.
The rules I would carry anywhere
- Dump with the database’s own tools, at its own version.
- Never mirror live database files and call it a backup.
- Copy offsite, never sync — a sync replicates deletions.
- At least one copy must be immutable, held by a credential that cannot delete.
- Restore on a schedule, automatically, and let it fail loudly.
- Alert on the absence of a recent backup, not on the failure of a job. A job that stops running
entirely never fails.