Across three physical sites there were flat allow rules letting every personal device reach
every piece of infrastructure. One of them had 6.3 million hits against it. Another, at a
different site, had 1.9 million. They were there because at some point somebody needed to reach
something, and a broad rule made that work.
I replaced them with a policy where people reach infrastructure over the VPN or not at
all, and declared the whole thing in code.
The shape
- Infrastructure networks mesh across all three sites over the site-to-site
link, so machines, cluster nodes and management networks behave like one place regardless of
which building they are in. - People networks reach exactly one thing on the infrastructure side: the VPN
endpoint at their own site, on one UDP port. Everything else goes through the tunnel, where
access is decided by identity rather than by which network jack you found. - People classes are siloed from each other. Wired and wireless personal
devices can reach each other; separate classes of people cannot. - Guest is fully isolated — gateway and internet only, nothing reaches in.
- A third party gets their own zone. One site hosts equipment belonging to a
different provider. That equipment now sits in a zone of its own that reaches itself, the gateway
and the internet, and nothing else — verified pair by pair in both directions.
A legacy remote-access VPN server, predating the site-to-site mesh and with exactly one peer
left, was turned off. Two ways in is one more than necessary.
The bug that had been there the whole time
The mesh was supposed to be complete and was not. Connections from the main site to two remote
management networks died, while every other pair worked.
The only hosts on those networks were the VPN routers for each site. Those routers had
accept-routes enabled, which installs the VPN’s routes into the operating system’s
routing table. So a connection arriving over the site-to-site link had its reply sent back out
through the VPN tunnel instead — asymmetric, and dropped.
Turning off route acceptance on the site routers fixed it. They still advertise their own
networks and still act as exit nodes; they just no longer route their replies into the tunnel.
There is a testing trap buried in this that cost me time. My own workstation has a VPN client
that does accept routes, so from there the remote networks were reachable — over the
tunnel, not over the mesh. Testing the site-to-site link from a machine on the VPN proves
nothing. I re-verified from hosts with no VPN client at all: nine out of nine pairs, in
both directions.
Breaking it during the change
Isolating the third party’s equipment went wrong in the middle. The tool deleted the old rule
that let their devices reach each other, then failed to move the networks into the new zone —
first rejected for writing read-only fields, then a server error on the retry. For a few minutes
their devices could not reach each other. Internet access was unaffected and nobody noticed, but
that was luck, not design.
The cause was doing it in two steps. Creating the new zone with the network
memberships moves them atomically; the tool now does only that, and the two-step path is gone.
Removing an allow before the replacement exists is a self-inflicted outage window, and it is
obvious in hindsight in the way these always are.
Why it is in code
All of this is now declared in a single tool that plans and applies against each site’s
console. It owns the people, guest and third-party policy, and it asserts the invariants that are
easy to undo by hand — that guest is isolated, that nothing else reaches the third party’s zone,
that people can still reach the things they are supposed to.
A clean plan against all three consoles is the proof. The next person to “just add a rule real
quick” gets a diff instead of a silent divergence.
The honest limit: about a third of the policy surface is declared. The rest of the zones are
still hand-managed, and adopting them is a dump-import-prove cycle per zone rather than something
to do in one pass. I would rather have a third of it genuinely under control than all of it under
a config file nobody trusts.
The part that is a people problem
The rule I removed had 6.3 million hits. Something was using it. After the change, personal
devices need the VPN to reach the hypervisors, the clusters, the media server and the LAN
services — which is the intent, and is also a real change in daily friction for everyone on those
networks.
Any exception that turns out to be genuinely needed goes back in the declaration, not into the
console by hand. Otherwise the file stops describing the network, and a policy file that is not
true is worse than no policy file at all.