At a cloud-native software company I was the sole platform engineer for a microservice estate
spread across four cloud projects. Fifteen services, fifteen separate deployment pipelines, each
grown independently and each slightly different from the others.
A full release took most of a working day. Not because any single step was slow, but because
fifteen different processes had to be driven in the right order by someone who knew the
differences between them. That someone was increasingly me, and that is a bad place for an
organisation to be.
Rebuilt as one trunk-based pipeline, a full rollout took thirteen minutes.
What actually made it fast
The speed was a consequence, not the goal. The goal was making the fifteen pipelines into one
thing, and almost all of the time saving came from that rather than from optimising any
individual step.
Reusable pipeline libraries. Build stages became containerised and
parameterised — one implementation, called fifteen ways, instead of fifteen implementations that
had drifted. A fix to a build stage became a fix everywhere rather than a fix in one place and a
note to do the rest later.
Trunk-based development. Long-lived branches were where the time actually went
— not in the pipeline at all, but in reconciling branches that had diverged for weeks. Shortening
the distance between a commit and production removed an entire category of work that had never
shown up on a pipeline dashboard because it happened in people’s heads.
Gates that block rather than report. Security and quality checks were already
running; they produced reports that were read when someone had time. Making a critical finding fail
the release changed what the pipeline was for. It stopped being a build system with advice attached
and became something the organisation could actually rely on to hold a line.
That last change is the one that needs the most care politically, because on the day you turn it
on, the pipeline starts telling people no. It is only viable if the findings are accurate and the
path to resolving one is short, and getting there took longer than the pipeline work.
Local environments that matched CI
A recurring category of failure was code that built locally and failed in the pipeline, which is
expensive precisely because it is discovered late and debugged remotely.
I wrote a command-line tool in Go that creates a standardised containerised development shell
with the environment variables and credentials already configured, so a developer’s local
environment matched what the pipeline would do. “Works on my machine” became a much rarer sentence,
and the remaining cases were genuine differences worth investigating rather than setup drift.
The work that was not pipeline work
Two things from that role that mattered as much and look less impressive on a diagram.
I designed the development environment structure so that access governance could be
enforced by the IT organisation rather than by me. Environment separation and role-based
permissions that IT could administer directly. Before that, access decisions routed through
engineering because only engineering understood the layout — which is a bottleneck dressed up as
security, and it fails the moment the engineer who understands it is on holiday.
I delivered self-service tooling that let non-engineering teams query and act on
operational data directly. The measurable outcome was removing engineering from routine
request handling. The unmeasured outcome was better: the people with the actual question stopped
having to translate it into a ticket and wait.
Alongside that: disaster recovery for the toolchain with backups automated to cold storage,
identity federation with enforced multi-factor across cloud accounts, and provisioning production
infrastructure as code with automated database snapshots and recovery orchestration.
What I would say about the number
Ninety-six percent is a real measurement and it is also the least interesting thing here,
because it mostly describes how bad the starting point was. Any estate with fifteen divergent
pipelines has a number like that available to whoever consolidates them.
The durable outcomes were that releases became boring, that a release stopped requiring a
specific person, and that the security posture moved from advisory to enforced. Thirteen minutes is
what those look like on a stopwatch.