The Routing Flag That Took Five Sites Down
A VPN option swallowed the container network on a shared web server. Five sites went down for twenty hours, and the guard that should have prevented it had been silently dead for two months.
Infrastructure · ReliabilityWhen the Failure Domain Was a Lie
Five storage pools were told to keep three copies of every write. They were keeping all three on the same machine.
Infrastructure · ReliabilityEvery Disk in the Organisation, on One Dashboard
Fifty-eight drives across four sites, one collector — written because the standard one reports no age at all for the drives that matter most.
Infrastructure · Reliability · Self-hostingEvery Log in One Place
Container logs, hypervisor syslog and firewall events from three sites, in one lake — and the sizing decision that keeps the loud feed from erasing the quiet one.
Infrastructure · NetworkingThree Clusters, One of Them on Cellular
Two remote sites got their own single-node Kubernetes clusters, managed from the main site — and the one on a cellular link taught me to count bytes again.
NetworkingA Network Where People Cannot Reach Infrastructure
Three sites, a software-defined mesh, and a zone policy that removed a rule with 6.3 million hits on it.
ReliabilityBackups Nobody Had Ever Restored
Every database now dumps itself with its own tools, ships two directions, and gets restored on a schedule — because the only backup that counts is one you have put back.
Agents & AI · Self-hostingSix Agents, One Channel Each
A fleet of specialist AI agents that could not talk to each other, and the database-level fence that finally let them.
Infrastructure · ReliabilityThree Ways Unlocked State Ate Live Infrastructure
The same missing safety mechanism destroyed live resources three separate times. Each time it looked like a different bug.
Infrastructure · Self-hostingTwenty Services, Self-Hosted
Ticketing, file sharing, meetings, search, backups, security monitoring and a container registry — replaced with software I own, in about six weeks.
Agents & AI · Infrastructure · Self-hostingA Browser Farm for Agents
Ten Chrome browsers running in Kubernetes, each with a real profile and a real login, so an AI agent can use the web the way a person does.
Infrastructure · ReliabilityNine Copies of Every Write
My Kubernetes control plane had been running five times slower than etcd considers healthy for over a year. I spent two hours blaming Ceph recovery throttles, caused an outage proving it, and then found…
Agents & AIThe Question That Vanished, and the Fix That Was Almost Right
A self-hosted AI agent crashed with "no user query found in messages" — on a request that demonstrably contained a user query. The cause was an upstream renderer that validates messages after the server…
Agents & AIWorking Inside a Small Context Window
An agent went quiet mid-conversation. The documented threshold was wrong, the intuitive fix was backwards, and the real constraint was a number nobody had added up.
Agents & AI · InfrastructureTwo Cards, One Model, and the Ethernet Between Them
Pooling two Tesla T40s under llama.cpp's RPC backend bought a quarter-million-token context window on hardware I own — and cost half the generation speed, because every single token had to round-trip over plain TCP…
Agents & AI · ReliabilityTurning a Document Archive into a Tool an AI Agent Can Use, Safely
A self-hosted document archive holds years of receipts, statements, and scans — useless to an AI assistant unless it can be reached safely from the outside. The build: an OAuth-gated bridge that took a…
Agents & AIA Local Voice Agent, and the Two Weeks It Took to Make It Fast
Getting a self-hosted AI agent to hold a real-time voice conversation over Discord — on our own GPUs, with nothing leaving the network — turned into a two-week latency hunt through cold GPU handles,…
Infrastructure · NetworkingThe Felt Experience of a Well-Tuned Network
Most networks are utilities. A few are tools. The difference is whether you can feel the friction every time information passes through. You spend eight hours a day inside your network. More, if you…
Agents & AIAgentic Web Scraping with Browser Automation
The Challenge I went into this project wanting to build an agentic agent that could scrape the web and populate web application forms with information. Traditionally, building a scraping agent is stopped by the…
Client work · InfrastructureFifteen Pipelines Into One
A full microservice estate rebuilt onto a single trunk-based pipeline, taking a release from most of a working day to thirteen minutes.
Client work · InfrastructureResearch Lab Infrastructure Modernization
Executive Summary Faced with 10‑year‑old Ivy Bridge servers, fragmented GPUs, and mounting energy bills, the lab launched a six‑month modernization from January to June 2025. The goals were to: Key results: Key metric Early 2025 June 2025 Δ Idle rack draw…
Want this kind of work on your team?
The same care goes into client work: written down, checked, and yours to keep.