Two remote sites each needed to run a handful of workloads locally: a tunnel connector, a file
share, a sync client, a load balancer. Previously each ran a small collection of hand-built virtual
machines and containers, all configured individually, all drifting.
Each now runs a single-node Kubernetes cluster, built entirely by automation, and both are
managed from the main site through a GitOps controller that syncs their manifests from one
repository.
One node, deliberately
The instinct is two nodes for redundancy. That is worse here, and the reason is specific: a
two-member consensus cluster stops the moment either member is unavailable. Two nodes on one
physical host gives you double the failure surface and none of the availability. Redundancy
requires a third vote, which requires a second physical machine, which neither site has.
So each site is explicitly one node, and the recovery story is “rebuild it from automation,”
which is true because the whole thing — the virtual machine, the operating system image pinned by
hash, the cluster install, the registration with the central controller — is a playbook. That is a
better guarantee than a second node that cannot actually save you.
The cellular site
One of the two sites has no wired broadband. Its connection is cellular, with about 84 ms of
latency, sessions that drop, and data that is genuinely metered. Everything about operating that
site is shaped by those three facts.
The link drops packets above roughly 1,360 bytes. That was not a guess — it
was diagnosed from a backup job that would run, transfer for a while, and stall indefinitely.
Large packets crossing the cellular tunnel were being dropped without the sender learning about it,
so the transfer hung rather than failed. Pinning a smaller maximum packet size on the node fixed
it.
That one is worth dwelling on because of how it presented: not an error, not a failure, just a
job that never finished. Every component reported healthy. A silent stall is much harder to
diagnose than a crash, and path-size problems are one of the few things that produce them reliably.
Every byte is a decision. The metrics agent at each site does not scrape
everything and filter later — it keeps an explicit list of the series worth sending. A single
Kubernetes endpoint produces tens of thousands of series, and shipping all of them over a metered
cellular link every fifteen seconds would be an expensive way to monitor a site with four
workloads.
Backups are local first. The file-sync tool ships changed files whole rather
than as diffs, so an archive that changes daily crosses the link in full every day. Local backups
at that site run weekly, with the remote copies handling the rest. Both retention layers count
backups rather than days, which matters because one site is seasonal and goes offline for months —
a time-based retention policy would age every backup out over a winter.
A drive each, and nothing on it is backed up
Each site’s cluster owns a physical drive directly — passed through to the virtual machine
rather than carved out of shared storage. It holds cluster volumes, plus a few quota-capped
directories served back to the hypervisor as a network share for installation images and local
backups.
Nothing on that drive is in any backup, and that is written down in capital letters in the
documentation, because it is exactly the sort of thing that is obvious when you build it and
invisible eighteen months later when someone stores something important there.
Moving a sync client without breaking it
The file-sync clients at both sites moved off the hypervisor and into the cluster. The reason
was embarrassing once I measured it: on the hypervisor, the sync client reached the site’s own data
drive over a network file share — served by a pod in the cluster on that same machine. Every byte
crossed the network gateway twice to travel a few centimetres.
The move itself had one hard constraint. A sync client’s identity is a certificate, and a new
one comes up looking perfectly healthy with every folder unshared. The peers see an
unknown device; nothing errors; nothing syncs. So the migration carries the existing certificate
rather than generating one, and the tooling deliberately fails on a missing identity rather
than minting a new one — because the failure mode of a fresh identity is silent and the
failure mode of a hard stop is a message on your screen.
The index was not carried; it is too large to move that way. Each instance rescans locally and
transfers nothing, because the files are already there. Peers saw an address change and nothing
else.
What the remote sites got
Before: a few hand-built machines per site, each configured by hand, each drifting, each a small
mystery when something broke.
After: one cluster per site, built by a playbook, with workloads declared in a repository and
synced by a controller. Both sites are also registered into central monitoring, with their own
labels and their own alert group, so a site going quiet is a thing that pages rather than a thing
somebody notices eventually.