Over roughly six weeks I stood up twenty self-hosted services on my own infrastructure, each
replacing something that is normally a subscription: ticketing, file sharing, URL shortening, RSS,
video meetings, backup management, file indexing, developer tools, a spatial workspace, malware
scanning, network speed testing, a source-control server and container registry, security
monitoring, a log lake, and a personal finance system.
The interesting part is not the list. It is that doing twenty of these in a row turns
self-hosting from a series of one-off projects into a pattern with a small number of recurring
decisions — and a small number of recurring traps.
The shape almost all of them take
Most of these are one pod containing the application plus its database as a loopback-only
sidecar. The database binds to localhost inside the pod and is reachable by nothing else. There is
no service, no network policy to write, and no password crossing a network.
That shape carries a constraint that has to be respected rather than worked around: one
writer. A single database process, a single search index, a single set of lock files.
Every one of these is deployed with a replica count validated to zero or one, and an update
strategy that stops the old pod before starting the new one. A rolling update would briefly run
two writers against one volume, which is how you corrupt a database while all your health checks
are green.
Where an application genuinely does have separable processes — a web tier, a worker, a scheduler
— that shape does not fit, and it gets real services instead. But a second scheduler double-fires
every recurring task, which in a finance application means duplicate transactions, so the
single-writer rule reappears in a different costume.
What is exposed to the internet, and what is not
Of twenty services, three are reachable from the public internet. The rest resolve only on the
local network or over the VPN.
The three that are public are public for a reason: a ticketing system whose users are outside
the organisation, and a URL shortener that has to answer anyone by definition. Everything else —
file sharing, meetings, dashboards, the file index, security monitoring — is something only I or
my systems need to reach, and there is no upside to it having a public address.
One of these moved from public to local-network-only three days after launch, when it became
clear that a file-sharing tool did not need to be on the internet to do its job. Shrinking the
exposed surface after the fact is unusual and it should not be. Most things grow their exposure
over time because nobody revisits the decision.
The traps that recurred
Pin the immutable tag. Several of these publish a version tag that gets rebuilt
weekly under the same name. Pinning what looks like a version pins nothing; the build tag is the
real identifier. The failure mode is a pod that comes back different after a restart you did not
think was an upgrade.
Applications read the host’s CPU count, not their own limit. More than one of
these started one worker process per core on a large machine — dozens of workers for a service
with three users — because the standard way to count CPUs reports the physical machine. Worker
counts are set explicitly everywhere now.
Health checks have a one-second default timeout. That is long enough until the
storage is briefly busy, at which point the check times out, the container is killed, and a
perfectly healthy service restarts in a loop. Worse, when the database is a sidecar, killing the
container takes the database down with it. Several apparent crashes were the health check doing
the crashing.
A loopback service cannot be checked over the network. Health checks are
performed by an agent that dials the pod’s address. A database bound to localhost never answers
that, so a network-based check on a loopback sidecar can never succeed. It has to execute a command
inside the container instead. This one restarted a completely healthy database every few minutes
until I understood it.
First-boot seeds go stale. Most of these create their admin account from
environment variables on first boot only. Change that password in the application’s UI and the
stored copy is now wrong, with nothing to tell you. Every one of them is documented as
first-boot-only for that reason.
Some keys can never be rotated. Several of these derive data-encryption keys
from an application secret. Rotating it does not log everyone out — it makes stored credentials
permanently unreadable. In the finance application that means every bank connection becomes
undecryptable. Those keys are marked never-rotate everywhere they appear.
Was it worth it?
Honestly: for a business, mostly no. Twenty services is twenty upgrade paths, twenty backup
plans and twenty ways to be woken up. The subscriptions exist because operating software is real
work, and paying someone to do it is usually correct.
Where it does pay is narrower and real. Data you should not hand to a third party. Tools whose
per-seat pricing scales with your headcount rather than your usage. Anything you need to keep
working when the vendor changes their pricing, their terms, or their mind. And — the reason I did
it at this volume — a working knowledge of twenty different failure modes, which is not something
you can read your way to.
The version of this I would recommend to a small team is three or four services, chosen because
the subscription genuinely hurts or the data genuinely matters, run on the same shape as each other
so that operating one teaches you how to operate the rest.