Half the drives in my fleet had no reported age, no wear level and no temperature. Not
wrong values — no values. And I did not know, because the dashboard showed the drives that
did report, and a missing row does not look like anything.
Why the standard collector was blind
The distribution-packaged drive-health collector reads a specific attribute number from the
drive’s attribute table to get power-on hours. That works on spinning disks and older solid-state
drives. An NVMe drive has no attribute table at all — it reports health through a
completely different structure — so the collector emits nothing for it.
On one host, five of eleven drives were invisible. Every one of those was either a storage
cluster member or a boot device: the drives whose failure matters most and whose age you most want
to know.
So I wrote a replacement. It parses the drive tool’s structured output, emits the same metric
names the standard collector does — deliberately, so one dashboard and one alert set cover both —
and adds the half it omits: NVMe power-on hours, wear and temperature. It now runs every fifteen
minutes on nine hosts across four sites, covering fifty-eight physical drives.
The traps, all of which cost real time
Never run two collectors on one host. The metrics agent rejects the
entire scrape when it sees a duplicate series — so overlapping drive collectors take out
CPU, memory and filesystem metrics too. The old collector is therefore masked rather than
disabled, because a package upgrade re-enables something that is merely disabled.
A drive attribute is 48 bits and vendors pack counters into it. One enterprise
drive reported power-on hours as 1,005,022,365,560 — about 114 million years. The vendor is storing
several counters in one field. Reading the tool’s rendered human-readable string and taking the
leading integer gives the right answer; reading the raw value does not.
Temperature needs two thresholds. Sixty degrees on a spinning disk means
something is wrong. On an M.2 NVMe it is an ordinary Tuesday — they throttle in the low eighties,
and one site’s boot drive sits at 64 °C all day quite happily. A single 60 °C rule went pending on
half the fleet within an hour of being written. Different physics, different thresholds.
A relabel looks exactly like a vanished disk. The rule that catches a drive
falling off the bus compares against a two-hour-old sample. Adding a new label to every series made
twelve drives look, to that rule, like they had disappeared and been replaced by strangers. Twelve
false alerts within a minute. The comparison now requires the label on both sides.
The alert nobody writes
Most drive alerting watches values crossing thresholds: wear climbing, errors growing, spares
depleting. All of that is there.
The rule that took longest to think of is the one for a drive that stops reporting
entirely. A drive that falls off the bus does not report a bad value — it goes quiet, and
every threshold rule in the world stays green, because a rule comparing a missing number to a limit
never fires.
This is the same shape as a backup job that stops running rather than failing, and a monitored
host that stops pushing rather than reporting unhealthy. In each case the alert has to be written
against absence, and absence is the thing you do not think to write a rule for, because
while you are writing it everything is present.
Four layers, because no single one sees everything
The dashboard has four sections, and the reason is that the obvious one has a hole in it:
- Physical drives — the fifty-eight, with age, wear, temperature and errors.
- Storage cluster members — each one showing the physical device beneath it, so
a degraded cluster member points at a drive rather than at an abstraction. - Storage pools — pool-level health on the machines that use them.
- Filesystems — all thirty-six, and this is the only layer that covers the
virtual machines, whose disks are virtual and have no drive health to report at all.
Without that last layer, eight machines are simply absent from a dashboard called “drive
health,” which is precisely the kind of gap that is invisible until it matters.
One charting note that cost an hour: the dashboard tool will not merge two result sets with
different label sets, and a field one set lacks reads as null rather than as “no opinion.” Every
storage cluster member rendered as three partly-filled rows until the queries were rewritten to
carry matching labels throughout.
Where it ended up
Forty-six drives monitored at one site, with no age reported for any NVMe, became fifty-eight
drives across four sites with complete health on all of them. The alert set went from eight rules
to thirteen, with the new ones covering the silent failures rather than the loud ones.
The shared web server — which had no monitoring agent at all, was in no scrape configuration,
and could not reach the monitoring network — joined for the first time, which is how I came to
notice that the sites on it had been logging a resource limit daily for months with nothing
watching.