From the sky to the bedrock.
Fifteen years across the whole stack — AI agents, applications, cloud platforms, networks and the hardware underneath. I write up what I build, including the parts that went wrong.
What I’ve been building.
Four of the last few months’ projects. Every one of them has a write-up behind it rather than a claim.
A Browser Farm for Agents
Ten Chrome browsers running in Kubernetes, each with a real profile and a real login, so an AI agent can use the web the way a person does.
Every Log in One Place
Container logs, hypervisor syslog and firewall events from three sites, in one lake — and the sizing decision that keeps the loud feed from erasing the quiet one.
A Network Where People Cannot Reach Infrastructure
Three sites, a software-defined mesh, and a zone policy that removed a rule with 6.3 million hits on it.
Fifteen Pipelines Into One
A full microservice estate rebuilt onto a single trunk-based pipeline, taking a release from most of a working day to thirteen minutes.
Enterprise SaaS, media and IoT, agencies, small business, nonprofits.
- IoT & enterprise SaaSTSI
- Media appliancesNorth Shore Automation
- Agency platform & hostingLuminFire
- Lake healthLittle Sand Lake Area Association
- Heritage nonprofitSteiger Heritage Club
Six things I keep ending up in.
Each topic is a set of case studies, not a service page.
What I’m building right now.
Short posts from the build log as the work happens, and long-form case studies once it’s done.
Build log Live from Mastodon
@jacob on feed.jacobnollette.com
- View on Mastodon ↗
Really want to use this at my next role:
https://www.infracost.io/ - View on Mastodon ↗
None of that was supposed to be about drives. But sizing from worst case meant measuring the drives properly, and measuring them properly is how I found that TRIM had never run on any of the Ceph flash: 15 of 16 devices had issued exactly zero discard commands, across roughly 13,700 power-on hours each. Ceph ships that setting off by default.
- View on Mastodon ↗
The Ceph PG count had been too low for months and I'd deferred raising it twice, because the nodes sat at 80% memory requested and a split needs somewhere to go. They were using 25%. Once the requests were honest the split just started. A fair number of things on a deferred list are not blocked by the thing the list says is blocking them.
Case studies
What the problem was, what broke, and what fixed it.
The Routing Flag That Took Five Sites Down
A VPN option swallowed the container network on a shared web server. Five sites went down for twenty hours, and the guard that should have prevented it had been silently dead for two months.
Infrastructure · ReliabilityWhen the Failure Domain Was a Lie
Five storage pools were told to keep three copies of every write. They were keeping all three on the same machine.
Infrastructure · ReliabilityEvery Disk in the Organisation, on One Dashboard
Fifty-eight drives across four sites, one collector — written because the standard one reports no age at all for the drives that matter most.
Software Donkey builds the agents and apps, and teaches small teams to do it too.
A standing team of six AI agents runs real work every day. The studio’s first apps are in private testing.
Let’s talk.
Whether that’s a role, a project, or a second opinion on something that keeps breaking.