The build log.
Short posts about work as it actually goes: what shipped, what broke, and what turned out to be wrong. Pulled live from feed.jacobnollette.com.
Updates every few minutes
- View on Mastodon ↗
Declared the people, guest and remote-access firewall policy for three UniFi sites in code tonight: people networks now reach infrastructure only over tailscale, guest networks are fully isolated, and an old WireGuard link is switched off. The SD-WAN review turned up exactly one broken path. The site tailscale routers accepted tailnet routes, so replies to connections that arrived over the SD-WAN left through tailscale and never came back. One flag per router fixed it.
- View on Mastodon ↗
Built softwaredonkey.com, the studio's homepage: one index.html, no build step, no dependencies, GitHub Pages, DNS as Terraform. The herd on it is the six agents from this fleet, each drawn on the same donkey geometry (ears, head, jaw, muzzle, eyes) with its own palette and one prop. Four were already their working Slack icons; the other two were drawn to match. https://softwaredonkey.com
- View on Mastodon ↗
Gave a remote site's k3s VM a whole 2 TB NVMe by PCIe passthrough, served it back out as SMB from inside the cluster, and retired the NAS it replaced. The first version kept the hypervisor's local backup copies on that share, served by a VM on the same host. Circular, and it lasted a day: the hypervisor now gets a hard-capped 500 GiB share for ISOs and templates, and that site backs up straight to the backup server. The drive is invisible to VM backups; what matters on it needs its own copy.
- View on Mastodon ↗
The three sites' networks read as one estate now: the remote UniFi consoles have Terraform stacks, each site's tailnet router covers every LAN, Rancher left the public tunnel, and the remote hypervisor consoles are served on the LAN by each site's own ingress. Rough edge: those certificates didn't issue for a day, because the DNS token they were planned around was never created. The local DNS records went live anyway, and a local answer beats the public one, so names that had worked broke.
- View on Mastodon ↗
The smaller site runs on a cellular uplink, and through the site-to-site tunnel it silently drops every packet over about 1360 bytes. PMTU discovery did not save TCP: SSH key exchange stalled, Ansible hung at Gathering Facts, installs died halfway. Setting the VM's NIC MTU to 1340 made the node advertise an MSS that fits, and the whole install then took a minute. Same link, different symptom: a QUIC handshake never completes there, so that site's tunnel connector speaks HTTP/2.
- View on Mastodon ↗
Brought two more Kubernetes clusters online, one at each remote site: single-node k3s, built end to end by Ansible from a checksum-pinned cloud image, deployed to from the home cluster's Argo CD and imported into the same Rancher. One node on purpose: each site is one physical host, and a two-member etcd is worse than one, since quorum needs both. Then retired the proxy containers and firewall VMs they replaced. One of those firewalls turned out to have been routing nothing at all.
- View on Mastodon ↗
Storage slow-op alerts on two hypervisors read like dying disks. SMART was clean. The hosts were RAM-overcommitted: a 220 GiB Kubernetes worker VM plus a control-plane VM on 252 GiB of RAM, beside seven Ceph OSDs, swapping onto the same QLC drive that holds etcd. The slow disk was swap, and the same swap had been stalling etcd. The worker VMs were shrunk to 190 GiB, one GPU-passthrough node at a time, drain and cold stop each.
- View on Mastodon ↗
Why a sidecar and not exec into the database container: in there the database is PID 1. A Postgres postmaster that reaps a child it didn't start treats it as a crashed backend and sends every connection through crash recovery. Reproduced it while building this. A backup job should not be able to crash the thing it is backing up.
- View on Mastodon ↗
Rebuilt database backups across the cluster. Before, there was one real logical dump; everything else was crash-consistent copies, including a file mirror copying live database files mid-write. Now each database dumps nightly from a sidecar running its own image, goes offsite twice (one copy under 30-day object lock, uploaded by a key that cannot delete), and a monthly job restores the newest dump into a throwaway server, because an unrestored backup is a hope.
- View on Mastodon ↗
Added a security agent that reviews camera alerts. Its conversation model runs in the cloud and must never receive pixels, which is harder than it sounds: the runtime hands a thread's images to the first turn, and tool results carrying images go straight to the model. So image input is text-only, vision is a local 9B model on a P4 that returns a description, and the camera tool can post a snapshot for a human but only ever tells the model "posted". The fence is around the model, not the person.
- View on Mastodon ↗
Found that our NVR had recorded for four weeks and detected nothing: 1.3 million segments, zero events. The config never set detect.enabled, and this release defaults it to false. The pod also lacked the GPU runtime class, so its GPU slice came with no driver libraries, and the detector was a CPU model beside an idle card. The tell was an inference speed of exactly 10.0, the placeholder before any inference has ever run. It now detects with YOLOv9 on two Tesla P4s.
- View on Mastodon ↗
The agent fleet is six now, one per domain: administration, facilities, maintenance, health, finance, security. New rule: one agent per Slack channel, and every agent is ambient in its own rooms and transcribes voice. The old mention-gating and "only one of you transcribes" rules existed only because two agents shared rooms. Separate rooms deleted the rules. They reach each other through a shared store of conclusions and handoffs instead, fenced by Postgres row-level security per agent.
- View on Mastodon ↗
Developer boxes and agents are StatefulSets with updateStrategy OnDelete now: an apply stages a change and restarts nothing, and a person rolls a box when nobody is sitting in it. Two surprises. kubectl rollout restart under OnDelete mints a new revision and replaces no pod, which looks exactly like it worked. And status.currentRevision stays pinned to the old revision forever; compare the pod's controller-revision-hash to updateRevision instead.
- View on Mastodon ↗
The fix made it worse first. Purging the superseded driver generations left the kept packages marked auto with nothing depending on them, and the next apt autoremove deleted the entire driver from two nodes. A survey found the kernel, headers, iSCSI and container toolkit marked auto on every node, safe only by coincidence. Each node group now declares what autoremove may never take, marks it manual (not held, which blocks security fixes), and fails the play if a dry run would still take one.
- View on Mastodon ↗
Unattended upgrades moved the NVIDIA userspace on three GPU workers. That does not set reboot-required, but it invalidates the loaded kernel module, so every GPU container fails to START with "driver/library version mismatch" while the running ones carry on. It surfaced a day and a half later, as an unrelated pod that could not schedule anywhere. Every GPU workload on those nodes was in the same state and simply hadn't restarted yet.
- View on Mastodon ↗
Stood up a LAN-only Jitsi and every call joined with no audio or video. The bridge's UDP service was externalTrafficPolicy: Local, but the L2 load-balancer speakers only run on the control-plane nodes and the bridge ran on a worker. No node that could answer ARP had a local endpoint, so media black-holed while pages, signalling and health checks were all green. The tell was total_ice_succeeded stuck at 0. Also: two participants go peer-to-peer, so a real bridge test needs three.
- View on Mastodon ↗
Three days on a Terraform 403 against the tailnet policy: "calling actor does not have enough permissions". The OAuth client had write on ACL and DNS the entire time. The provider block asked for acl:read and dns:read at token exchange, so a write-capable client minted a read-only token. A raw API POST with the same credentials returned 200. That one-line test would have ended it on day one.
- View on Mastodon ↗
A merge touching an unrelated namespace made CI re-apply a shared stack from main. main still declared GitLab CE 18.9.1. The live instance was EE 19.3.1, upgraded from a branch. Downgraded four minor versions onto a database volume already a Postgres major ahead, it could not start at all, for six and a half hours. GitLab has its own namespace and state now, but the real lesson is the branch: a live upgrade that isn't on main is a rollback waiting for a merge.
- View on Mastodon ↗
The last storage VM is gone. What used to be a hand-built NAS is now one Kubernetes pod: CephFS roots mounted straight in, rclone and LucidLink as privileged sidecars pushing their FUSE mounts into a shell that can actually read them, and an smbd-only Samba sidecar serving subtrees of the same CephFS. No Service, no load balancer: the pod carries its own WireGuard identity, so a Mac anywhere on the SD-WAN mounts smb://storage like it's on the LAN. Software-defined storage, end to end.
- View on Mastodon ↗
The other lesson from the same night: a from-source build of the PBS packages died at the CI job's 1h default while still unpacking apt. dpkg fsyncs every file, and the build's overlay sat on consumer NVMe with no power-loss protection, so 55 minutes bought 228 packages. force-unsafe-io in the build stages and a real timeout; build containers are disposable.
Want this kind of work on your project?
The same care goes into client work: written down, checked, and yours to keep.