Infrastructure-as-code tools keep a state file describing what they believe exists. Most
backends support a lock, so two runs cannot write that file at once. The object-storage backend I
use does not.
I knew that. I had read it. I did not understand what it meant until it cost me live
infrastructure three times in a week, wearing a different disguise each time.
One: thirty-one browser resources vanished
Two runs overlapped. One had a slightly older picture of the world. It finished last, wrote its
state file over the other’s, and thirty-one live objects simply stopped being in the state.
Nothing errored. Both runs reported success — they each did exactly what they were told, against
a world that changed underneath them. The resources were still running in the cluster and no longer
existed as far as the tool was concerned, which meant the next run would try to create them fresh
and collide with the ones already there.
Two: a half-applied fleet destroyed
The same week, a concurrent run caught a fleet mid-creation and destroyed the half that
existed. Some resources were created by the first run, not yet in the state the second run had
loaded, and therefore — from the second run’s point of view — unmanaged leftovers to be removed.
A tool doing precisely what it was designed to do, with a stale picture, is indistinguishable
from a tool malfunctioning. That is what makes this class of bug so disorienting.
Three: continuous integration downgraded a production server
This is the expensive one. About six hours of downtime.
Our automation applies from the main branch on merge. I was upgrading a source-control server
on a feature branch — the upgraded version was live, working, and declared only on that branch.
Then an unrelated change, touching a completely different part of the system, was merged to
main.
The merge triggered an apply from main. Main still declared the old version. The
automation dutifully reconciled reality to match, and downgraded a running server by a major
version — across a schema that the older version could not read.
Nobody did anything careless. The upgrade was tested. The merge was unrelated. The failure is
structural: if your automation applies from main, then anything live that main does not
declare is scheduled for deletion, and the trigger is somebody else’s unrelated merge.
That one has cost me twice more in smaller ways: a continuous-integration run reapplying main
and sweeping resources that existed only on an unmerged branch. Same mechanism, less blast
radius.
What changed
Split the state. Each namespace, each domain, each stack has its own state file
and its own automation. The failures above all involved a shared state file where unrelated work
overlapped. Separation means an unrelated merge cannot reach into something it does not own — which
is exactly why the source-control server now lives in its own namespace with its own state, rather
than sharing with a dozen other services.
Merge the configuration before you apply it anywhere. This inverts how I used
to work: prove the change on a branch, then merge. Now, for anything the automation will reconcile
from main, the declaration lands on main first. Applying from a branch creates live resources that
main does not know about, and every one of those is a deletion waiting for an unrelated commit.
Check before applying by hand. Look for a run in flight; look at what other
working copies are holding. The checks are trivial and the cost of skipping them is measured in
hours.
Serialise the automation. Concurrency controls per workflow, so two applies
against the same state cannot start.
The part that generalises
These three incidents read, in the moment, as three unrelated problems: a provider quirk, a
race condition, an upgrade gone wrong. They are one problem. Every one of them is “two writers,
one state, no lock.”
The reason it took three to see it is that the symptom is always local and always specific —
you debug the resources that disappeared, not the mechanism that let them. I wrote all three up,
and it was only reading them next to each other that the shape came out. That is a large part of
why I keep an engineering changelog at all: individual incidents teach you individual lessons, and
a pile of them occasionally teaches you something none of them contained.