Five WordPress sites on one shared server started returning 502s. They stayed down for about
twenty hours. Nothing had been deployed, nothing had been upgraded, and the server was up the
whole time.
What happened
The server runs each site’s PHP in containers behind a web server, which means the sites talk
to each other over the container bridge networks — private address ranges that only exist on that
machine.
That host also joins our VPN with route acceptance enabled, so it can reach internal services.
One of the routes advertised on the VPN is a broad private range, and it is broad enough
to contain the container bridge subnets.
When that route won, traffic destined for a container on the same physical machine was sent
into the VPN tunnel instead. It went looking across a wide-area network for an address that
existed six inches away, found nothing, and the web server returned 502 for every request.
The fix is a routing-policy rule per bridge subnet, at a priority above the VPN’s, pinning
those destinations to the local routing table. Two rules. The sites came back immediately.
The part that actually matters
This machine already had a guard for exactly this. A system service had been installed months
earlier to assert those rules at boot.
It reported active. It had reported active for two months. The rules it was
supposed to install were not there.
It was a one-shot service marked to stay active after exiting. That flag makes the service
report success forever after a single successful run — which is fine if nothing ever removes its
work. But the VPN client re-evaluates and rewrites routing state on reconnect, on network change,
on restart. Every one of those quietly undid the guard, and the guard, having already run, never
noticed and never ran again.
So the monitoring view was: service enabled, service active, no errors, no alerts. And the
thing it existed to guarantee had not been true since roughly the week it was installed.
What replaced it
The new guard is deliberately shaped so it cannot decay the same way:
- It does not stay active. It runs, does its work, and exits. There is no
“already done” state to get stuck in. - A timer re-asserts it every few minutes, so anything that removes the rules
has a bounded window before they come back. - Its check asserts on the routing table, not on the service. The verification
step reads the actual rules. “The unit is active” is not accepted as evidence of anything.
That third point is the generalisable one. A health check that asks a component whether
it ran is not a health check. It has to ask the system whether the effect is present.
The two answers agreed for months and then disagreed for months, and only one of them was ever
true.
The fix I proposed first was wrong
My first instinct was to narrow the advertised route so it no longer overlapped the container
ranges. I was ready to make that change when I actually read the network definitions and found
that one of our real internal networks sits inside the same range as the container bridge. The
overlap is structural, not accidental. Narrowing the advertisement would have broken a working
network to fix a different one.
I also tested the idea of accepting only a few specific host routes instead of the whole range
— injecting single-address routes for just the services this machine needs. It does not work. The
VPN client routes by its own internal peer map, not by the operating system’s table, so the route
installs cleanly, looks correct, and the packets still die. I confirmed that on a spare host before
proposing it anywhere.
Both of those were an hour each, and both were worth the hour, because the alternative was
shipping a confident fix that would have broken something else.
Where it leaves things
The routing rules are now deployed by configuration management across the whole fleet rather
than existing on one machine, and the same playbook removes the old decaying guards wherever they
were installed.
The real conclusion is architectural: route acceptance is all-or-nothing, and a machine that
only needs to reach four services should not be accepting a route to everything. The direction
this is going is that those four services each become addressable on the VPN in their own right,
so the permission lives in an access policy rather than in a routing table on every host. The day
the routing guard can be deleted is the day that migration is finished.
Twenty hours is a long time, and it was long because nothing was watching
those sites closely enough to page anybody. That gap got closed alongside the fix: the fleet now
pushes metrics and logs to central monitoring, and web-application metrics — the ones that would
have shown PHP workers saturating — exist on those machines for the first time.