A replicated storage cluster’s core promise is that a copy of your data lives on more than one
machine, so losing a machine loses no data. That promise is enforced by a placement rule, and the
rule has a setting for what counts as a separate failure domain: a separate disk, or a
separate host.
Five pools in my cluster were set to disk.
Every one of them was faithfully keeping three copies of every write, and nothing prevented all
three from landing on three disks inside the same server. Lose that server — a power supply, a
mainboard, a kernel panic during an upgrade — and every one of those three copies goes at once.
How it stayed hidden
This is not a configuration that degrades. It reports perfect health.
The cluster said all placement groups were clean, all copies present, all pools healthy. It was
telling the truth about the wrong question. It was verifying “are there three copies?” — yes — and
never asking “are they anywhere useful?”, because that was never what it was told to check.
The pools were years old. They had been created before the cluster had enough hosts for
host-level separation to be meaningful, when disk-level was genuinely the only option, and nobody
revisited it once more machines arrived. That is the ordinary way this happens: a correct decision
for a smaller system, silently inherited by a larger one.
The fix is not just the rule
Changing the placement rule moves data, and on a live cluster that is a rebalance to be
scheduled rather than a switch to flip. But the placement rule was only half of it.
The other half is the minimum number of copies required to accept a write. Set too low, the
cluster keeps accepting writes while dangerously under-replicated — which feels like resilience and
is actually the cluster agreeing to become your single point of failure. Set correctly, it stops
accepting writes and tells you, which is the behaviour you want even though it is the behaviour
that wakes you up.
Both were corrected together, because fixing placement without fixing the minimum leaves a
cluster that spreads copies properly and will still happily run on one of them.
What else this cluster taught me
The same cluster produced a run of failures over a few months that were each instructive:
An interrupted delete can break every listing in a pool. One orphaned entry in
a deletion queue made the storage library fail while enumerating — so listing the pool, listing the
queue, and the hypervisor’s own storage listing all broke at once. Every clone, snapshot and
restore then timed out on a lock. One stuck entry, a whole pool’s management surface unusable, and
the fault was in a structure most operators never look at.
Consumer drives cook under recovery load. Raising recovery throughput to make a
rebalance finish sooner pushed a set of drives into thermal throttling, which made everything
slower than before the change. Recovery tuning on consumer hardware is a thermal problem disguised
as a throughput problem.
The storage cluster was not the bottleneck; memory was. A node running both
storage daemons and a large virtual machine was over-committed on memory and swapping — onto the
same device holding the cluster database. The symptom was storage slow-operation warnings, and the
cause was swap. I spent real time tuning storage before measuring memory, which was the wrong order
and is the order most people take, because the alert names the component that noticed rather than
the component at fault.
A configuration class mask can resolve correctly and never apply. Settings
applied to a class of daemon read back exactly right when queried and were not in effect on the
running process at boot. I had verified by asking the configuration system what it thought, which
is not the same as verifying behaviour. Per-daemon settings and an assertion against a runtime
effect replaced it.
The thread through all of it
Every one of these is a system confidently reporting health against a question that was not the
one that mattered. Three copies, on one host. A configuration that resolves, and does not apply. A
clean rebalance on a node that is swapping.
The habit I took from it is to ask, when something reports healthy, what exactly did it
check? — and to make the verification step assert on an effect in the world rather than on a
component’s opinion of itself. It is slower to write. It is the difference between a monitor and a
mascot.